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ABSTRACT 


This research investigated how different experts in a single domain chose their 
individual subjective evaluation criteria of a highly aggregate task based upon their 
individual differences. The Conning Officer Virtual Environment (COVE) was utilized 
to provide a domain of experts and a subjectively evaluated task. 116 expert ship- 
handlers were investigated to understand how their personality affects their evaluation of 
a novice performing an underway replenishment (UNREP). The experts were issued a 
survey that inventoried their personality, UNREP evaluation criteria, and ship handling 
style. In general, the participant experts were lower in Neuroticism and higher in 
Extraversion and Conscientiousness than the average adult. Extraversion appeared to be 
correlated with the expert’s desire to use Sensory Input as a critical evaluation criterion 
(po = .18) while Openness was correlated with Analytical Input (p = .16) and UNREP style 
(9 = .16) as cnitical evaluation factors. Also correlated with UNREP style was 
Agreeableness (p = .16). Finally, the expert’s level of Conscientiousness correlated with 
the critical evaluation criteria of Analytical Input (p = .17) and Sensory Input (p = .39). 
Results from this research provide insight to the link between observed behavior and its 
subjective evaluation and will allow COVE’s programmers to develop an Intelligent 


Tutoring System (ITS) that will customize the automated training process. 
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[.. INTRODUCTION 


A. MOTIVATION 


The military services have historically been an apprenticeship system. Beginning 
with the early days of sail, United States Naval Midshipmen would spend several years 
serving aboard a single ship with a single captain. Under the tutelage and guidance of his 
master captain, the apprentice midshipman would learn the art of sail and war. A 
midshipman would be promoted to the rank of an officer of the line only after gaining his 
captain’s full trust and confidence in his knowledge and abilities. This type of highly 
specialized training required vast resources and the dedication of numerous personnel. 

Whether a sailor, marine, soldier, or airman, the United States Military warnor of 
the new millennium is required to do more with less. Fewer troops, weapons, training 
time, and fiscal resources are requiring the armed forces to re-evaluate every facet of 
operations. In particular, budget draw downs and the increasing complexity of hardware 
necessitates the need to create cost-effective training alternatives. As computing power 
and speed increase, the desire for utilizing computers as a beneficial training tool also 
increases. Using modern computers’ ever increasing high fidelity virtual environments 
(VEs) as a training tool for performing spatial and cognitive tasks are a particular area of 
increasing demand since VEs provide the potential capability for a trainee to practice and 
master complex and highly dangerous tasks safely, efficiently, and economically 


[CAIR96]. 


B. OBJECTIVE 

‘The best known generalization in human learning ts that practice makes perfect” 
[ANNES89]. The caveat to this cliché is that the student is practicing the nght task in the 
nght ways. The ability to properly react to any situation requires expert guidance and 
proper intervention at critical points during training. Without a good teacher, practice 
alone is not always enough to become competent at a complex task. Even more 
devastating is the possibility that the student will get worse at the learned task and 
experience a negative training transfer [BOLD87]. 

While VE is a relatively new training tool, a VE training system is not 
pragmatically different from any previous generation of training tool. VE training must, 
just like any other training system, provide students with the skills and knowledge 
required to meet the demands of the trained task and the needs of the overnding 
organization [CAIR96]. While any form of training requires several key components to 
be effective, one of the most essential steps to developing a successful training program is 
providing quality feedback via instruction and evaluation. 

The Conning Officer Virtual Environment (COVE) Ship Handling Trainer is one 
example of a VE that provides an economically attractive alternative to traditional 
methods of practicing ship handling while providing an integrated means of instruction. 
COVE, currently under development by Naval Air Warfare Training Systems Division 
(NAWC-TSD), simulates ship-handling scenarios where the trainee 1s immersed in a VE, 
complete with an integrated intelligent tutoring system (ITS), in the form of a simulated 
interactive commanding officer. COVE is a flexible and portable unit that is intended to 


build and reinforce ship-handling skills with minimal requirements for instructor 
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intensiveness and costly ship resources [MEAD99]. If the deployed implementation of 
COVE is successful, Junior Officers (JOs) will have an opportunity to develop basic 
skills and practice difficult scenarios in a controlled environment without the need to 
have entire ships at sea, saving time, dollars, and possibly even lives. 

‘Providing the trainee with knowledge of the results is one of the most common 
training program interventions and one which is generally believed to have a powerful 
effect on learning” [ANNE89]. In the case of COVE, quality feedback to the trainee 
requires the ITS to be more than a scripted set of rules. The ITS must provide immediate 
guidance and feedback that is accurate and meets accepted standards, just as a 
Commanding Officer (CO) would at sea, else msk the loss of valuable training time and a 
possible negative transfer training experience [TENN99]. 

ITS feedback should both conform to accepted, safe practices and the 
requirements of the JO’s CO. However, the dynamics of handling a ship at sea combined 
with individual differences of COs makes it difficult to have a single standardized set of 
feedback responses. Just as the onginal shipmasters trained their apprentices uniquely, 
today’s COs train their JOs according to their predilections. Different COs will have 
different benchmarks based upon their own style of expertise, expemences, and 
personality, resulting in different COs evaluating the same evolution differently [NPS99]. 
In order to gain maximum benefit for the fleet, COVE’s ITS must be flexible enough to 
meet the needs of the different fleet experts. 

While a prime example, COVE is just a single example of a trainer that requires 
extensive knowledge and that has infinitely many ways to arrive at a "correct" solution 


that is "correct" only in the eyes of the evaluator. Topics easily range from driving ships, 


to land navigation, to philosophy. Essentially, anything that involves asynchronous 
student-paced instruction and training of a highly complex aggregate task that involves 
subjectivity in evaluation can benefit from the relationships explored by the COVE ITS 
and student. 

This study investigates how simulator performance evaluation should be modeled 
based upon the personality composition of the evaluating expert. With respect to the 
COVE trainer, it is desired to understand the different evaluation criteria used by different 
COs and its relationship to their personality, ultimately resulting in a more accurate ITS 


where the Virtual CO (VCO) could approximate a wide range of real world COs. 


To be true to form, one would have to have the many different styles of 
COs within the system and the ability to choose which one you need. At 
the one end would be the screamer that we may be most familiar with who 
will throw you off the bridge if you go too far, and at the other end would 
be the true mentor who lets you get to the point of no return only to help 
you avoid the collision that you thought was inevitable. 

Commanding Officer of an LPD 


This accurate modeling results in more effective trainer time by teaching the JO the same 
lessons his real world CO would teach, increasing the effectiveness and overall benefit of 


the trainer. 


Ge THESIS QUESTIONS 


The following questions are addressed tn this thesis: 


e Is there a relationship between one’s personality and one’s 
expertise? 


e Ifsuch a relationship exists, can it be quantified? 


e Does it extend beyond individual expertise to the expert’s 
evaluation of others’ performance? 


e What is the range of characteristics of different ship driving styles? 


Addressing these questions is the first step in building a more accurate ITS for 
COVE. Since this research is only the initial exploration between human behavior and 
expert evaluation, it is intended to begin the initial com atone a database for the 
COVE ITS. Understanding the answers to the aforementioned questions will provide 
COVE’s ITS programmers with a realistic model to base various prototypical VCOs 
upon. Furthermore, these answers also lay the foundation for automating the relationship 
and increasing the fidelity between instructor and student in any VE with an ITS. This 
added insight will help mate the ITS with the student, potentially increasing positive 


training transfer for any VE training system. 


D. APPROACH 


In order to answer the questions posed by this research, knowledge about the 


relationship between experts and novices is required. Along with the nature of expertise, 


= 


knowledge about personality and its measurement must be obtained. Furthermore, the 
scope of this work requires an understanding of the sociological domain within which the 
experts and novices reside. 

Naval officers achieve the prestige of command only by displaying sustained 
superior performance, primarily at sea. As the senior ship driver aboard, and the one 
person ultimately responsible for any mishap, the Commanding Officer (CO) is the 
resident ship-handling expert. How a ship is dnven by any of the ship’s officers is a 
direct statement about the ship handling abilities of its CO. 

Few evolutions make or break a CO’s reputation like the UNREP approach to the 
auxiliary replenishment ship since the approach is a calling card for the CO’s style and 
ability. While all UNREP experts achieve the same ultimate end goal of coming along 
side the replenishment ship, different COs accomplish this task differently. Some prefer 
to “John Wayne” with large speed differentials and small distances from “rubbing paint” 
while others prefer more of a slow and steady approach. Some COs base decisions on 
aggregate big picture data while others require more specific input. 

While the UNREP is one of the greatest showcases of skills for the surface 
warfare officer, it is also one of the most dangerous where the potential for loss of life 
and damage to not only one but also two ships is extremely high. The ability to actually 
practice this formidable task at sea is limited and can quickly evolve into a situation too 
complex for a junior officer to handle. These criteria result in good VE training being 
crucial and indicate that UNREP is an excellent VE candidate since it allows the 
opportunity for officers to develop prerequisite skills in a safe and controlled environment 


with minimal operating cost. 


Because of all of the aforementioned factors, UNREP was the vehicle for this 
research and analysis. Its importance in the sociological domain of the ship driver also 
makes it suitable to determine the correlations with ship driver personality. An analysis | 
of the expert evaluation of an UNREP approach as performed by a less-experienced JO 
was correlated with the personality of the expert to answer the thesis questions. Results 


can be directly applied to COVE’s existing ITS during COVE simulated UNREPs. 


E. SUMMARY OF CHAPTERS 


The remainder of this thesis is broken down into the following chapters: 


e Chapter II provides background information on the mental and 
behavioral processes invoked during UNREP and other complex 
tasks. First, a review of the basic components of an UNREP is 
performed followed by a summary of COVE and ITS previous 
research. Next, a more in-depth view of the differences between 
experts and novices is explored in order to understand the 
differences between COs and JOs. Additionally, individual 
differences and their effects on decision making are explored. 
Finally, personality and its measurement are discussed in order to 
understand how individual expert COs are different from each 
other. 


e Chapter III discusses the apparatus utilized to gather information 
for this research. Reasons for selection, design, and development 
are covered for the two data collection tools, the NEO-FFI 
personality inventory and the Ship Handler Evaluation Survey. 


e Chapter IV delineates the methods utilized for data collection and 
analysis. An explanation detailing the administration of the survey 
is provided in addition to a summary of the construction of the data 
package. 


Chapter V summarizes the results from the data collection and 
analysis. Results are provided detailing the personality 
characteristics of the participant expert ship handlers, the cntical 
evaluation criteria utilized by expert ship-handlers for evaluation of 
novice JOs, and _ significant correlations observed between 
personality and critical evaluation criteria. 


Chapter VI presents a final discussion of the results of this thesis 
and describes areas requiring further research. Answers to thesis 
questions proposed by this research are addressed 1n addition to the 
possible ramifications of this research. 


Il. BACKGROUND 


A. THE UNREP EVOLUTION 


The UNREP evolution, while complex and dangerous during execution, 1s a 
particularly straightforward task. Two ships, an approach vessel and a replenishment 
vessel execute the evolution. The approach vessel is a warship that requires 
replenishment of its fuel and or stores. The replenishment vessel is usually a refueling 
tanker. The overall goal of the evolution is for the approaching vessel to come within 
close proximity of the replenishing vessel and bring on fuel and other supplies with out 
any damage to personnel or equipment. 

The UNREP is composed of distinct phases consisting of preparations, waiting, 
approach, alongside, and breakaway. Figure 1 is a diagram depicting the different phases 
involved in a plausible UNREP scenario and highlights some of the distances involved 
between the two ships participating in an UNREP. The evolution actually starts hours 
before the actual transfer of supplies 1s executed by performing the preparation phase. 
Checks of ships systems and a pre-execution brief are performed on both ships to ensure 
that both the ships and crews are prepared to perform the actual task. 

The next phase, the waiting phase, is just prior to the commencement of the 
approach. Duming this phase, the approach vessel maneuvers to a waiting station where 


the approach vessel will perform its last checks and wait for a signal from the 


replenishing vessel to commence the approach. The waiting station is usually an area 


approximately 1000 yards astern of the replenishment vessel. 
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Figure 1: A Diagram of a possible UNREP evolution. 
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Once both ships are on an agreed upon course and speed and are ready, signals are 
made and the approach phase commences. Dumning the approach phase, the approaching 
vessel maneuvers from waiting station to a position directly alongside the replenishment 
vessel. It is during the approach phase that the first interaction of physical forces occurs 
between the two vessels. 

Once the approach is made, lines connecting the two ships are secured and the 
approach phase transitions into the alongside phase. During the alongside phase, the 
approach vessel maintains a constant position -latiee to the replenishment vessel dunng 
the transfer of fuel and stores. Radio communication is maintained between both vessels 
during the alongside phase until transfer 1s complete between the replenishment vessel 
and the approach vessel. Transfer time primarily depends upon the amount of supplies to 
be transferred, but typically is less than an hour. 

Once all supplies are transferred, all connecting lines between the two ships are 
cast off, marking the beginning of the breakaway phase. During this final phase, the 
approach vessel maneuvers away from the replenishment vessel. Once clear of the 
replenishment vessel, the approach vessel 1s no longer restricted in its ability to maneuver 


and can proceed on its own independent course and speed. 


B. THE CONNING OFFICER VIRTUAL ENVIRONMENT AND 


INTELLIGENT TUTORING SYSTEM 


The COVE trainer is a direct evolvement of a previous project by NAWC-TSD, 


the Virtual Environment Submarine (VESUB) Simulator. WESUB was intended to 


Ll 


provide a means for submarine officers to practice surfaced submarine evolutions, in 
particular transit in and out of port, with out the need of a surfaced submarine. COVE 
combines some of the onginal VESUB visual simulation architecture with voice 
recognition and an integrated intelligent tutoring system. Ideally, COVE is a portable, 
low cost training solution that provides the user with a high fidelity synthetic ship driving 
experience and requires no operator monitoring or intervention [MEAD99]. 

Most previously implemented expert systems possess limited capability for 
diagnosis and feedback making them relatively unsuitable for training purposes 
[TENN99]. In order for an artificially intelligent (AJ) training system to be successful, it 
must possess the capability to learn from experience by making human-like associations 
requiring a sense of appropriate output and understanding of needs, desires, and emotions 
[DREY96]. A possible architecture that meets these criteria incorporates adaptive 
technology into a pedagogical agent. An example of a first generation ITS is STEVE 
(Soar Training for Virtual Environments), which is currently under development by the 
Air Force Research Laboratory [TENN99]. STEVE is designed to be a modular agent 
implementation for the purpose of instruction in a variety of computer-based learning 
environments [JOHN98]. 

The COVE trainer incorporates an ITS in the form of a Virtual Commanding 
Officer (VCO). The VCO is a pedagogical agent that instructs the JO on how to properly 
drive the ship during a ship handling evolution such as UNREP [TENN99]. Previous 
research investigated three possible profiles for the VCO consisting of a passive VCO, a 
proactive VCO, and an aggressive VCO. The classification of passive, proactive, or 


aggressive was based primarily on a CO’s predilection to recommend course and speed 
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changes while a JO was conning the ship during an UNREP evolution. The three 
classifications were chosen for their broad coverage of different ship driving styles and 


were intended to represent the majority of ship drivers in the fleet. 


Cc. DIFFERENCES BETWEEN EXPERTS AND NOVICES 


‘*... want the JO to learn to drive the ship the way I drive the ship.” 


Member of SURFLANT staff addressing 
NAWC-TSD about COVE and ITS. 

While a CO will usually find other styles acceptable, he will prefer his JOs to 
drive the ship in a manner similar to his. Much like a father teaching his teenager how to 
drive a car, the expert CO will first instruct, and later expect the novice JO to analyze 
data and make decisions in the same fashion as the CO. These expectations are the basis 
for the expert CO’s evaluation of the novice JO and are shaped by the CO’s expertise. 

UNREPs are dynamic, complex tasks and UNREP skill cannot be neatly 
categorized under a single type of expertise. Expertise itself is diverse and is segregated 


into the four following categories: [CHI88] 


e PRACTICAL EXPERTISE that pnmarily deals with motor skills 
or mental skills. Examples of practical skills are typing, 
memorizing restaurant orders, or mental calculation. This type of 
expertise often allows for parallel thought processing. 


e PROBLEM SOLVING EXPERTISE requiring specific domain 
related knowledge. Examples of problem solving expertise are 
computer programming or solving physics problems. 
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e ILL-DEFINED EXPERTISE that requires decisions under 
uncertainty, such as when some uncontrolled intervening event 
occurs between the choice and the outcome. An example of an ill- 
defined problem is predicting stock market performance [CHI88]._ 

e DIAGNOSTIC EXPERTISE where metacognition is required to 
accurately access the reason for a given circumstance or set of 
facts. An example of diagnostic expertise is properly accessing an 
illness or medical condition from x-rays or symptoms. 

All four of the previous expertise categories apply to an UNREP evolution. 
Commands are issued and executed with practical expertise. Estimating times and speeds 
in Open-ocean utilize problem-solving techniques. Given the dynamic nature of an 
UNREP due to the uncontrollable forces of nature and the interactions of two separate 
independent ship drivers simultaneously, UNREPs require both ill-defined and diagnostic 
expertise. A novice JO must demonstrate proficiency of all types of expertise in order to 
receive a favorable evaluation of the UNREP evolution from his CO. 

Expert ship handlers usually distinguish themselves from novices by determining 
the quickest, most efficient courses of action, and when a ship handling evolution is 


getting out of control. In distinguishing themselves from novices, experts, regardless of 


the area of expertise, share common traits. These commonalties are summarized as: 


e Experts excel mainly at their own domains. 

e Experts perceive large meaningful patterns in their domains. 
e Experts quickly solve problems with little error. 

e Experts have superior short-term and long-term memory. 


e Experts see and represent a problem at a deep (more principled) 
level. 
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e Experts spend a great deal of time analyzing a problem 
qualitatively. 


e Experts have strong self-monitoring skills. 


These traits usually result in an expert performing a task quicker and with fewer errors 
[CHI88]. 

Even though experts distinguish themselves apart from novices in common ways, 
there are still wide variabilities amongst the experts themselves. These variabilities are 
unique to each expert and are often referred to as individual differences. These 
differences influence how the expert responds to situations, teaches his novices, and 
evaluates his trainees’ proficiency. Understanding individual differences of the CO are 


critical inputs to producing useful feedback for the JO. 


D. DECISION-MAKING AND INDIVIDUAL DIFFERENCES 


A study performed by the U.S. Army Research Institute for the Behavioral and 
Social Sciences assessing how senior Army officers made critical battlefield decisions 
discovered that not all experts analyze situations and make decisions the same way 
[COHE96]. Most experts generally fall into two completely different paradigms. Some 
experts follow an analytical approach where decision-making is characterized by 
attempting to use rational and computational methods. In contrast, a recognition expert 
would attempt to make decisions based on fitting the situation into a known pattern and 


responding with a familiar label or plan of action. 
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Another U.S. Army Research Institute for the Behavioral and Social Sciences 
study investigated the effects of expertise, cognitive style, and mission on what 
information is used by senior Army officers during tactical decision making in an attempt 
to develop a tactical decision aid [MICH88]. Their research indicated that a tactical 
decision aid must be adaptable to individual differences such as personality, cognitive 
style, and preferences for sensory modality and communication mode. These findings 
correlate with the research on how Army officers performed under stressful situations. 
Their findings showed that personality exhibited some consistent patterns of response to 
stressful situations. Their research assumed that there is a reciprocal causality between 
individual, situational, and response variables and that the way an individual responds to 


a situation is directly affected by the individual’s personality. 


E. THE FIVE FACTOR MODEL OF PERSONALITY AND FEEDBACK 


The UNREP is an extremely stressful evolution for both novice and expert. Since 
an expert CO will perform an UNREP based upon his expertise paradigm, which is 
shaped by his individual characteristics, it is necessary to study the COs personality. 
Personality is often explained by the Five-Factor Model (FFM), which describes 
personality in terms of five distinct personality traits. The ‘‘ FFM onginated in initial 
works by Fiske (1949), Norman (1963), and Tuppes and Christal (1963), who produced a 
highly stable structure with five factors” [SALG97]. The FFM is extremely attractive due 
to its empirical roots. While most models are derived from theoretical perspectives, the 


lexical FFM has a theoretically neutral position [WIDI97]. 
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The Revised NEO Personality Inventory (NEO PI-R) is modeled after the FFM. It 
is a widely accepted measure of personality developed by Dr. Paul Costa and Dr. Robert 
McCrae, assesses personality in terms of Neuroticism, Extraversion, Openness, 
Agreeableness, and Conscientiousness. The five personality factors are described in the 


following way: 


e EXTRAVERSION is the factor that describes people who are rated 
by their peers as “‘sociable, fun-loving, affectionate, fnendly, and 
talkative” [MCCR87] versus “reserved, timid, and quiet” 
[SALG97]. 


e People high in AGREEABLENESS are forgiving, lenient, 
sympathetic, agreeable, and softhearted, according to peer ratings 
[MCCR87]. Peers describe those low in Agreeableness in more 
negative terms: ruthless, uncooperative, suspicious, and stingy. 


e Peers describe people high in CONSCIENTIOUSNESS as careful, 
well organized, punctual, ambitious, and persevering [MCCR87]. 
Conscientiousness “includes both proactive (hardworking, 
ambitious) and inhibitive (dutiful, scrupulous) aspects” 
[MCCR89]. 


e People who score high on NEUROTICISM typically report 
negative emotions such as worry, insecurity, self-consciousness, 
and tempermentalness [MCCR87] whereas people with low 
Neuroticism are calm, self-confident, and cool [SALG97]. 


e The final factor in this model is OPENNESS. Adjectives from 
lexical studies that descnbe this factor include “onginal, 
imaginative, broad interests, and daring” ,[MCCR87]. “Openness 


defines individuals who are creative curious, and cultured versus 
practical with narrow interests. [SALG97]” 


The five factors of personality have implications for occupational performance 
and therapy. Most research studying the relationship between personality and job 


performance only attempts to correlate quality of job performance with personality 
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[CLON96]. Two meta-analytic studies by Barrick and Mount (1991) and Tett and 
Jackson (1991) find that Conscientiousness is the only predictor of quality of job 
performance [RUST99]. 

There 1s less research on the link between individual differences and method of 
task completion. A previous study showed correlation between individual differences 
and variability in expertise [NPS99]. Specifically, ship handler’s methods for performing 
an UNREP varied into two distinct categories, analogical or analytical. Whether a CO 
performed an UNREP in an analytical fashion, or an analogical fashion could be 
correlated with the CO’s personality and cognitive style. A study investigating the theory 
that personality is more differentiated at higher levels of ability discovered that some 
personality traits are statistically more variable for individuals at high versus low levels of 
ability [AUST97]. This research also showed relationships between types of judgment 
and FFM factors and Intelligence Quotient (IQ). 

Rust, 1999, investigated the ability of the FFM to predict supervisor’s ratings of 
performance. In his research, Rust administered the Orpheus; broad-spectrum work 
based personality questionnaire to employees. His findings showed a correlation between 
the FFM results of the self-evaluation Orpheus and appropriate supervisor ratings. In 


evaluating the FFM within the context of work based behavior: 


e High extroversion people are generally happier working with 
others while low extroverts tend to prefer work requiring 
independence. 


e High agreeableness results in individuals with a desire for a more 


cooperative, problem-solving approach the lower Agreeableness 
results in an ability to make tough decisions. 
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e People displaying high openness to experience seek alternative 
solutions and desire different methods, while low openness to 
experience individuals desire traditional approaches and respect 
established values. 


e Low neurotics tend to disregard feelings of others, perform better 
under stress, but tend to lack caution. 


e High conscientious people tend to excel at detailed tasks, but may 
become over involved in minutiae while low conscientiousness 
people have little patience for mundane tasks, and prefer to see the 
big picture. 

[RUST99] 

This research is important because personality is a factor for how a CO learns, and 
subsequently trains. The expert is more inclined to use teaching techniques in a manner 
that he understands the best. Previous research has attempted to correlate personality 
traits with various learning styles. This research indicated that Extroversion and 
Agreeableness are linked with more active types of learning [FURN96]. Therefore, it is a 
conclusion that Extroversion and Agreeableness could explain active forms of teaching. 
In the case of an UNREP, these personality traits could explain why some COs are more 
actively involved with the JO dunng the evolution than others are. 

Salgado analyzed three prior meta-analysis studying the relationship between 
personality and job criteria. In general, Salgado discovered that “Extraversion is a valid 
predictor of training proficiency (p = .26), as are Neuroticism (p = .07), Agreeableness 
(9 = .10), and Openness to Experience (9 = .25)” [SALG97]. Furthermore, personality 
compatibility between teacher and student will potentially affect the teacher’s evaluation 
of an evolution. Research demonstrates that students achieved higher levels in classes 
when teacher-student personality compatibility is high [FURN96]. This teacher and 
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student dynamic should also apply to VE training, and if properly modeled, will further 


increase the accuracy of the training feedback. 


ke PERSONALITY MEASUREMENT 


The assessment of personality is a major application of psychology to real world 
concems and is extremely varied in its administration and utilization. Clinical 
psychologists evaluate a patient’s personality in an attempt to determine if the patient 
possesses abnormal symptoms or feelings. A school psychologist will assess a child’s 
personality in order to determine any causes of possible learning or adjustment problems. 
Counseling psychologists attempt to determine the best job for a particular person by 
matching the individual’s needs and interests with the requirements of the position. 
Finally, research psychologists assess the personalities of experiment participants to 
account for experimental behavior or correlate personality characteristics with other 
measures [SCHU90]. 

Regardless of the end goal, some assessment techniques are more objective while 
some techniques are wholly subjective and prone to bias. The best techniques possess 
standardization, reliability, and validity. Standardization insures consistency and 
uniformity of the procedures utilized for the test administration. Reliability insures 
consistency of results to the assessment device. Finally, validity insures that the test 
device results are an accurate measurement of the intended measured variables 


[SCHU9O]. 
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Multiple methods exist to assess personality. A primary method is referred to as 
the self-report inventory method. In the self-report method, people report on themselves 
by answering questions about their feelings and behavior in a variety of simulated 
situations. The person taking the test must indicate how closely each item describes their 
own characteristics or how much they agree with each item. In general, self-report 
personality assessment methods are high in reliability and validity due to the standardized 
nature of administration, scoring, and evaluation of the results [SCHU90]. 

The Myers-Briggs Type Indicator (MBT]I) is a self-report personality survey 
created by Katharine Briggs and Isabel Bnggs Myers in the 1920s. The MBTI is based 
upon Carl Jung’s model of personality and is the primary method for measuring Jungian 
personality types. The MBTI measures introversion and extroversion and is used for 
research purposes as well as career counseling. The MBTI requires several hours to 
administer and evaluate and requires a trained psychological professional to interpret the 
SCOres. 

The Minnesota Multiphasic Personality Inventory (MMPI) is another frequently 
utilized personality assessment tool. The MMPI determines personality traits of 
hypochondniasis, depression, hysteria, psychopathic deviate, masculinity-feminity, 
paranoia, psychasthenia, schizophrenia, hypomania, and social introversion. The MMPI 
is primarily used by clinical psychologists as a diagnostic tool for assessing personality 
disorders, but is also utilized as a vocational tool. Unfortunately, like the MBTI, the 
MMPI is extremely long to administer and requires special training to interpret the 


results. 
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Projective testing methods are pnmarily utilized for assessing disturbed 
individuals. When presented with an ambiguous stimulus, such as a Rorschach inkblot, 
the patient will project personal needs, values, and fears onto the stimulus description. 
Projective techniques suffer from low reliability and validity due to the subjective nature 
of the result evaluations [SCHU90]. 

Behavioral assessment procedures evaluate a person’s behavior to a specific 
situation. Researchers assessing the personality of an entire group of people primarily 
utilize this method. For example, hospital staff will routinely observe patient behavior in 
order to identify behavioral trends in patients. This method requires specifically trained 
observers and is highly subject to observer bias, resulting in lower reliability and validity 


[SCHU90}. 
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Il. APPARATUS 


Ae NEO FFI 


The modeling of an expert CO response to an UNREP evolution is theoretically 
possible if the individual differences of each CO can be ascertained. Collecting accurate 
data about individual differences of COs requires selecting the correct personality 
assessment tool. Most COs are limited on time and relatively unsupportive of academic 
endeavors that take away from their operational duties. While the assessment tool must 
be highly reliable and valid, because of the population being examined and the purposes 
of this research, it must also be easy to administer, easy to complete with minimal time 
requirements, and easy to evaluate with little training required. 

While the MMPI is a predominantly used objective test for assessing personality, 
it is primarily used for assessing personality disorders [BERN94]. The MBTI is a widely 
utilized personality inventory implemented in career related management, but requires a 
trained psychological professional to administer and 1s too time intensive for the purposes 
of this research [SCHU90]. Though there are a number of objective tests designed to 
measure a broad range of personality variables in a normal population, an increasingly 
popular choice 1s the Neo Personality Inventory (NEO PI-R) [BERN94]. The NEO PI-R 
1s a prime choice for inventorying an expert’s personality since it is the predominant 
measure of the five factor model of personality [WIDI97]. The NEO PI-R consists of 240 


statements to which a person indicates an extent of agreement on a 5-point scale. The 


22 


NEO PI-R is often referred to as a lexical five factor model since it attempts to define 
personality in natural language terms. 

The “majority of academic psychologists, increasingly favor the NEO PI-R for 
assessment and research” [FURN96]. Furthermore, substantial research exists regarding 
NEO PI-R reliability and validity. Most important, the NEO PI-R has demonstrated 
consistent convergent and discriminant validity, as well as indicating how alternate 
models can be understood from the perspective of the five factor model [MCCR89]. 
Multiple studies have correlated established measures with the NEO PI-R to establish 
overlap, including the Eysenck Personality Inventory and Myers-Briggs Type Indicator 
[FURINGS): 

The creators of the NEO PI-R re-evaluated the usefulness and applicability of 
their personality assessment test. Their findings indicated “far more evidence of its 
comprehensiveness, universality, and practical relevance today than when the NEO-PI 
was first published” [MCCR97]. The NEO Five Factor Inventory (NEO-FFI) is a bnef 
60-question subset of the full 240 question NEO PI-R. The NEO PI-R’s additional length 
allows for more precise measurement and better false answer detection while the NEO- 
FFI shorter length accommodates a quicker administration time for the participant. Since 
the creators of the NEO PI-R do not envision any significant changes in the structure of 
the NEO PI-R in the near future, it is a logical conclusion that there are no major 
revisions planned for the NEO-FFI. 

The NEO-FFI scales show correlations with the NEO PI-R ranging from .75 to .89 
for each of the five factors. As subsets of the NEO PI-R domain scales, the NEO-FFI 


scales carry portions of the demonstrated validity of the full scales. While the NEO-FFI 
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scales are not equivalent to the full scales of the NEO PI-R, the shorter scales are 
approximately 85 percent as accurate as the full scales. In the case of the abbreviated 
scales of the NEO FFI, some precision is traded for speed and convenience [MCCR92]. 

Previous research inventoried five senior US Navy Surface Warfare Officers for 
individual differences to determine variability in personality using the NEO-FFI [NPS99]. 
Participants consisted of five Unites States Navy Commanders, military pay grade of O- 
5, all of which have been designated Surface Warfare their entire careers. Four of the five 
had served in Executive Officer positions as their last sea going billet, and one had served 
as a Commanding Officer. All five participants scored in the low category for 
Neuroticism with little variance. On average, the participants were high in Agreeableness 
and Extraversion and average in all other categories. The participants exhibited large 
variances in Openness and Agreeableness scores. 

This previous research justified the choice of the NEO-FFI as the tool to assess 
the personality of expert ship drivers [NPS99]. All participants clearly understood the 
standardized directions and had no questions. During the pilot experiment, the inventory 
was easy to administer and on average took less than 10 minutes to complete. Evaluation 
of the results required minimal time and were very easy to interpret by the researcher who 


had no formal personality assessment training. 


B. EXPERT SHIP HANDLING SURVEY 


As part of developing initial profiles for a VCO for COVE, a ship handling 


background questionnaire was utilized. While the questionnaire was primanily for 


25 


demographic purposes, it did attempt to elicit participant opinions about how to train a 
junior ship driver. The questionnaire was combined with ship driver interviews in order 
to determine the basic VCO framework [TENN99]. 

Previous research surveying and analyzing ship-driving Commanding Officers 
highlighted traits and characteristics that are the same as the traits exhibited by experts 
[TENN99]. Based upon this commonality, experienced ship-drivers were identified as 
experts. Expounding upon the original research, this investigation also utilizes a survey. 
In the early phases of research development, 1t was determined that a large population of 
expert participants was desired to ensure that the full spectrum of ship dnving styles was 
approached. Unfortunately, because of logistical constraints, any form of physical 
interview was impractical. Therefore, the investigator decided to utilize a survey for the 
primary method of ship driving style elicitation. 

Survey questions were primanly based upon previous research examining 
individual differences and ship driving style [NPS99]. In the previous research, the 
participating expert ship handlers were administered an open ended interview regarding 
UNREP. The expert participants were encouraged to state what the key factors were 
when they evaluated novice ship handlers. The results of these expert interviews build 
the fundamental core of this research’s survey. 

However, utilizing a questionnaire for the ship driving style elicitation posed 
challenges that required significant consideration. Since the survey was to be performed 
remotely by the ship handling evaluator participants, the survey must be extremely clear 
since the researcher would not be present to make any clarifications. The size of the 


population of expert ship-handlers precluded qualification of a participant as an expert for 
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reasons other than experience and position. Also, the questionnaire had to be concise 
since most expert ship handlers have limited time to diverge from their duties of running 
a warship. The goal was to maintain the expert ship handling survey completion time to 
under 15 minutes. 

The survey format primarily utilizes multiple choice or rating questions to elicit 
the desired information from the participant. Rating questions were specifically chosen 
because they produce an actual or absolute value of the trait being measured. This 
required developing the rating scale with equal intervals with an anchor position. These 
traits result in rating questions being easier to write and prone to fewer errors [GAOQ93]. 

Figure 2 details the ratings utilized for the survey. The rating scale was 
specifically developed to minimize respondent error and bias. The list of possible choices 
was set at five since most respondents can only distinguish between five to nine items 
[GAOQ93]. Furthermore, the list was maintained short in order to reduce primacy and 
recency effects, effect where respondents are biased toward the last few items because 
those are freshest in memory of a long list of items. The list of possible rating responses 
was always presented in the same ascending order to facilitate proper understanding of 
each rating and help aid recall. 

Another primary concern with developing a question involves avoiding 
inappropriate questions. The expert ship-handling domain is extremely sensitive to 
perceived nght and wrong ways of doing business. For example, this requires avoiding 
questions that might require an answer that is directly contrary to guidance or doctrine. 


Regardless of whether or not the expert disagrees with doctrine, it would be socially 
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unacceptable for the expert to declare that he conducts business in a contrary manner. In 


general survey questions were developed to avoid the following questions that: 


e Are not relevant to the evaluation goals; 
e Are perceived as an effort to obtain biased or one sided results; 
e Cannot or will not be answered accurately; 


e Are not geared to the respondent’s depth and range of information, 
knowledge, and perceptions; 


e Are not perceived by respondents as logical and necessary; 
e Require an unreasonable effort to answer; 

e Are threatening or embarrassing; 

e Are vague or ambiguous; or 


e Are unfair 
[GAO93]. 





Not Applicable (N/A) -There is no need to perform this action. 





Applicable (A) - This is a relatively minor action with large room for 
variation of execution. 

Somewhat Important (Sl) - An action that must be performed to have 
a successful UNREP approach, but with some room for variation of 
execution. 







Important (l) — This action must be performed well to have a successful 
approach with little variation of execution. 







Critical (C) - It is impossible to successfully complete an UNREP 
approach without performing this action flawlessly. 





Figure 2: Rating scale utilized for the Ship Driving Style Survey 
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In addition to avoiding inappropriate questions, the questions themselves “‘must be 
direct, orderly, precise, logical, concise, and grammatically correct. They must have 


unity, coherence, and emphasis” [GAO93 ]. 


C. EXPERIMENT PACKAGE 


The experiment package consisted of a single survey consisting of five 


subsections: 


e Introduction and Background. 

e Participant Expert Ship handler Demographic Questionnaire 
e Personality Inventory 

e Expert Ship Handler Survey 


e Conclusion and Comments. 


The overall goal was to maintain the total completion time to less than 30 
minutes, because it was a reasonable amount of time to accommodate completion of the 
survey during a lunch hour or other mealtime underway or in port. Furthermore, 
anonymity and privacy were highly stressed to promote participation and help elicit 
higher quality responses. 

Two forms of the survey were created, an Internet based survey and a traditional 
paper based survey. The layout of the traditional paper based survey utilized Government 
Accounting Office (GAO) survey guidelines for design. For example, font size was 


maintained at 10-point type and text was arranged into two columns to promote ease of 
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reading [GAO93]. The survey was printed front to back to minimize the apparent size of 
the document, reducing the likelihood that a potential expert ship handler would not 
complete the survey due to time requirements. Furthermore, the survey was bound to 
improve appearance and better accommodate the participant. 

The Internet based survey was created in order to allow participants the 
opportunity to complete the experiment without having to return any package via the mail 
system. Content was exactly identical to the paper based version, however some 
formatting was changed to better accommodate viewing on a 17 inch computer monitor. 
The visual layout was optimized for an 800x600 pixel screen size. The Internet based 
survey also automated data collection by sorting the participants’ responses into a 
database, removing the possibility of any data corruption by the researcher. This version 
of the survey was created using Microsoft FrontPage 2000® web development software 
with all code generated into Hyper Text Markup Language (HTML). 

Prior to commencement of the experiment, two senior surface warfare officers 
stationed at Naval postgraduate School tested the Internet version of the survey for 
usability and functionality. One usability participant was a senior Navy Commander, 
military pay grade O-5, who had previously served in the position of Executive Officer of 
a warship at sea. The other participant was a senior Navy Captain, military pay grade 
O-6, who had previously served as a Commodore of a squadron of warships. Changes 
regarding content of the Internet version of the survey were also made in the paper-based 
version of the survey to maintain continuity between the two experiment forms. 

Every expert ship handler requested to participate in the expenment was mailed 


the following items: 
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e Cover letter requesting participation from either Commander 
Surface Naval Forces Atlantic Vice Admiral Giffin (see 
APPENDIX A), or Commander Surface Naval Forces Pacific Vice 
Admiral Moore (see APPENDIX B). 


e Addendum to instructions detailing how to complete the Internet 
based version of the experiment survey (see APPENDIX C). 


e <A paper based copy of the experiment survey (see APPENDIX D). 


e A pre-addressed return envelope. 


While every United States Navy surface warship possesses Internet capability while in 
port or at sea, every expert ship handler was given the choice of participating via either 
the Internet based or traditional paper based survey version. While possessing the 
capability, it might not be feasible for an expert ship handler to participate in the 
experiment electronically because of operational constraints or material maintenance. 
Furthermore, the dual method of participation accounted for problems with electronic 
participation and also helped prevent requested experts from not participating because 


they were not comfortable with the method of electronic participation. 
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IV. METHODOLOGIES 


A. EXPERT POPULATION CANIDATES 


Along with deciding what to ask, evaluators must decide whom to ask. 
The people questioned must have the information the evaluators seek, they 
must be readily identifiable and accessible, they must be willing and able 
to answer, and they must be representative of the population being 
measured. 


[GA093] 


Since the purpose of this research 1s to learn about the relationships between 
expert evaluators and novices performing a subjective task, the experiment obviously 
required experts in a position to evaluate novices. Because COVE 1s the vehicle for the 
research, the target population of this research was the expert ship-handlers in a position 
to train and evaluate junior novice ship-handlers. 

Even though the CO is the ultimate person responsible for all ship operations and 
sets the tone for the conduct of all operations, he is not necessarily the only instructor and 
evaluator. In most circumstances, even during an UNREP, the Executive Officer (XO) 
also plays a vital role in instruction and evaluation of junior novice ship-handlers by 
augmenting the CO as an additional coach or evaluator. While not as common, a 
Department Head (DH) is an occasional additional mentor to the junior ship-handler and 


sometimes provides input to the CO for evaluation of the JO. 
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The experiment primarily targeted a sample of COs and XQOs currently serving 
aboard warships 1n the fleet. DH survey participation was also accepted if the DH was a 
recognized ship-handling expert by his CO or XO. These experts were selected because 
they are currently performing the analyzed task and are most familiar with the current 
doctrine and equipment utilized in the fleet. In addition to being a CO, XO, or DH, the 
expert candidate must be serving on a class of warship that conducts UNREPs as the 
approach vessel. This resulted in the exclusion of small craft such as mine hunters and 
coastal patrol craft. Furthermore, while tankers occasionally perform an UNREP as the 
approach vessel, non-Navy personnel who might not possess the same background as the 
targeted expert population usually operate them. 

Based upon the class of ship criteria, there were a combined total of 171 eligible 
warships between the Atlantic and Pacific naval forces. Since every US Naval warship 
has both a CO and an XO, there were a total of 342 potential ship-handling experts to 
sample from. While a larger sample size of ship-handling experts will be a better 
approximation of the total population of ship-handling experts, a minimum of 30 
experiment participants 1s required to satisfy the Central Limit Theorem statistical rule of 


thumb [DEVO95]. 


B. SURVEY ADMINISTRATION 


Experiment packages were assembled and mailed via United States Post Office 
First Class delivery to all 342 ship-handling expert candidates. The candidates were 


allowed approximately three weeks until the beginning of July 2000 to complete either 
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the web based or paper based survey. The administration period was selected to 
accommodate mail time both to and from the ship as well as an adequate time to review 
and complete the survey. 

If the web-based version was completed, the experiment participant was instructed 
to not return the paper-based version. If the participant completed the paper-based 
version, the data was recorded utilizing the web-based survey after venfying that the 
paper-based survey was not a duplicate of an already submitted web-based survey. This 
manual conversion of survey format was performed to accommodate automatic data 


collation and analysis. 


C. ANALYSIS 


All raw survey results were compiled into a single Microsoft Access® database. 
The database file was then exported into a Microsoft Excel® spreadsheet. Once in Excel 
format, the raw personality scores for Neuroticism (N), Extraversion (E), Openness (O), 
Agreeableness (A), and Conscientiousness (C) were computed. 

Questions from the ship-handling evaluation section of the survey were classified 


as one of six distinct types consisting of: 


e INTERACTION questions measuring the expert ship-handler’s 
preference for interaction between the novice JO and other entities. 
Other entities could consist of the other members of the bridge 
team, the replenishment ship, or the expert ship-handler himself. 


e ANALYTIC INPUT questions measuring the expert ship-handler’s 
preference for the type of rational based decision information that a 
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junior novice ship-handler should use. Examples of Analytic Input 
are rules of thumb, numerical data from ship’s sensors, and 
standard operating procedures, 


e SENSORY INPUT questions measuring the expert ship-handler’s 
preference for the type of instinctual based decision information 
that a junior novice ship-handler should use. Examples of Sensory 
Input are visual approximations for range, non-numenical or non- 
calculation based rules of thumb, and kinesthetic approximations 
for weather forces. 

e COMMUNICATION questions measuring the expert ship- 
handler’s preference for the type of communications the junior 
novice ship-handler should use. Examples of Communication are 
internal and external communications circuits. 

e UNREP STYLE questions measuring the general expert ship- 
handler’s approach to UNREP and what he expects of the junior 
novice ship-handler. UNREP Style questions also include the 
expert ship-handler’s general interpretation of UNREP guidance 
and doctrine. 

The average response for each of the six groups was calculated for each expert 
ship-handling participant. All data was then converted into an input file for the ARC 
software package, a menu driven statistical analysis tool developed at the University of 
Minnesota for applied regression. 

Once the data package was loaded into ARC, a statistical summary of the data 
package was created. The statistical summary contains information such as mean values, 
minimum and maximum values, median values, and standard deviations. Furthermore, 
the statistical summary contains a matnx of correlation values between the different 


variables of the data package. It 1s from this matnx that significant correlations were 


retrieved for discussion. 
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V. RESULTS AND DISCUSSION 


A. PARTICIPANT DEMOGRAPHICS 


Of the 342 ship-handling experts polled, 136 experts participated in the survey, of 
which 35 participated via the Internet. Eight surveys were incomplete and not used in the 
data analysis. One survey was completed by a participant who did not meet the criteria of 
a ship-handling expert as defined for this experiment. Eleven surveys were retumed too 
late to be included in the data analysis package. The resulting data analysis package 
consisted of a total of 116 surveys. 

At the time of the survey administration, 65 survey participants were serving in 
the CO billet and 48 were serving in the XO billet, and 2 were serving in the DH billet. 
Of all participants, only two ship-handling experts are female. Table 1 further 
summarizes some demographical information of the 116 analyzed ship-handling expert 
participants. 

On average, the ship-handling expert participants had served under eight different 
COs during their career. Furthermore, the average participant had performed between 50 
and 100 UNREPs during their career with a single ship-handling expert who had 
performed over 300 UNREPs. Eight participants were aviators and have not been surface 


warfare qualified for their entire careers. Major ship classes represented consisted of: 
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e Ajrcraft carriers, including both nuclear powered (CVN) and non- 
nuclear powered (CV) types, 


e Guided missile cruisers (CG), 


e Destroyers, including both guided missile (DDG) and non-guided 
missile (DD) types, 


e Guided missile frigates (FFG), 

e Transport ships including amphibious assault ships (LHA/LHD), 
dock landing ships (LSD), tank landing ships (LST), and 
amphibious transport dock ships (LPD), 


e Other warships not classified above. 


Figure 3 is a histogram that delineates the frequency of ship types for the 
participating expert ship-handlers. Figure 4 is a histogram that describes the participant 


billet distribution between observed ship types. 
Average Minimum Maximum Standard 
Observed Observed Observed Deviation 
ee =| 
Lieutenant . 
Rank Commander Captain N/A 
Commander 
Years Of Service 
Years Of Service | 


Table 1. Demographic Information for Ship-Handling Expert Participants. 
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Figure 3. Frequency of Participant Ship-Handling Expert Ship Types 
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Figure 4. Frequency of Billet Distribution Among 
Participant Ship-Handling Expert Ship Types 


B. PERSONALITY INVENTORY RESULTS 


Table 2 provides a statistical summary of the observed expert ship-handler NEO- 
FFI results. Figure 5 highlights the personality differences between the average 
participant expert ship-handler and the average over 18 year-old adult participants as 
defined within the NEO-FFI manual [MCCR92]. Means and standard deviations are 
contrasted for each personality trait. Notable differences are evident in Neuroticism, 


Extraversion, and Conscientiousness. 


} ! : 


Openness Agreeableness | Conscientiousness 


1 
i 


| Neuroticism Extraversion 








| (E) | (Oum! (A) | (C) 
Average! 10.91 34.69 | 27.36 j; 32.67 | 38.66 
| Min | — 0.00 15.00 | 15.00 | 13.00 | 24.00 | 
| Max | 31.00 46.00 | 43.00 4600 | 48.00 : 
Mode _—-—-6.00 33.00 | 27.00 3400 | 36.00 
Median —‘10.50 35.00 27.00 | 33.00 | 39.00 


Table 2. Statistical Summary For Participant Expert Ship-Handler NEO-FFI Results 


Table 3 delineates the personality differences amongst the participant expert ship- 
handlers. Personality scores were relatively consistent with the exception of carner 
expert ship-handlers. On average, carrier ship-handling experts are. significantly lower in 
Neuroticism and higher in Openness and Agreeableness. These personality differences 
could be related to the demographic difference between carer ship-handlers and other 
surface ship-handlers since all carmer COs and XOs are aviators. The only other 


demographic group to show slight deviations from the whole was the cruiser ship- 
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handling experts who typically scored slightly lower in Neuroticism and higher in 


Extraversion than the average participant expert ship-handler. 
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Figure 5. A Comparison of Observed Expert Ship-Handler Means 
and Standard Deviations to Typical Adults As Defined By The 
NEO-FFI Professional Manual. 





| CG | CVICVN | DD/DDG FFG Transport” 

| Neuroticism | O56 6.60 10.053 13296 ies 5 | 

| Extraversion | 37.00 | 5920) | 34.33 33.04 33.69 

| Openness | 26.42 | 32.40 | 27.80 26.96 26S 
Conscientiousness 40.11 42.20 38.69 | 38.11 37.23 | 





Table 3. Mean Participant Expert Ship-Handler NEO-FF] 
Results For Each Major Participant Ship Class 
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In order to compare a participant’s raw personality score to an average distribution 
of adults, the raw score is converted into a standardized classification group. The 
standardized personality classification groups consist of very low, low, average, high, and 
very high. Ranges for each of the classifications depend upon which score is being 
classified. Table 4 contains the standardized values for each personality classification 


range as defined in the NEO-FFI manual [MCCR92]. 


| Neuroticism Extraversion Openness. Agreeableness Conscientiousness 





VeryLow | 0-6 o=ns Picanto 240m |e 0-25 





Table 4. NEO-FFI Standardized Raw Score Classification Ranges 


Figure 6 1s a histogram depicting the frequency of Neuroticism scores amongst the 
participant expert ship-handlers. The distribution 1s positively skewed indicating that a 
majority of the participant expert ship-handlers possess low Neuroticism. Figure 7 1s a 
histogram that illustrates the distribution of Neuroticism classifications amongst the 
participant expert ship-handlers. Figure 7 confirms that 97% of the participants possess 


average or lower than average Neuroticism as defined in Table 4 [MCCR92]. 
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Figure 6. Frequency of Neuroticism Raw Scores 
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Figure 7. Distribution of Standardized Neuroticism Scores 


Figure § 1s a histogram highlighting the distribution of participant expert ship- 
handler Extraversion scores. Figure § illustrates a negatively skewed distribution 
indicating that most participants possess large scores for Extraversion. Figure 9, a 
histogram delineating the breakdown of the Extraversion raw scores, communicates that 
most participant expert ship-handlers were higher than average in Extraversion. 
Approximately 80% of the participant expert ship-handlers possess a high or very high 
Extraversion personality characteristic. 

Figure 10 is a histogram that displays the frequency of the observed Openness raw 
scores. Unlike Extraversion and Neuroticism, the participant expert ship-handlers appear 
to possess a symmetric distribution of Openness personality characteristics. Figure 11, 
the distribution of Openness classifications amongst the participants, appears to center 
around the average with an approximately normal distribution. 36% of all participant 
expert ship-handlers possess an average Openness personality characteristic while only 
28% possess a low characteristic and only 22% possess a high Openness characteristic. 

Figure 12 is a histogram illustrating the frequency of Agreeableness raw scores. 
Similar to Openness, Agreeableness also appears to be symmetrically distributed amongst 
the participant expert ship-handlers. However, as Figure 13 details, the majority of 
participants possess an average level of Agreeableness while only 27% possess a lower 
than average level and only 32% possess a higher than average Agreeableness personality 


trait. 
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Figure 8. Frequency of Extraversion Raw Scores 
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Figure 9. Distribution of Standardized Extraversion Scores 
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Figure 10. Frequency of Openness Raw Scores 
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Figure 11. Distribution of Standardized Openness Scores 
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Figure 12. Frequency of Agreeableness Raw Scores 
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Figure 13. Distribution of Standardized Agreeableness Scores 


Figure 14 1s a histogram that displays the frequency of Conscientiousness raw 
scores. Conscientiousness appears to have a slightly negatively skewed distnbution 
amongst the participant expert ship-handlers. Figure 15 showing the distribution of raw 
score classifications for Conscientiousness shows that 88% of all participants possess an 


average to verv high Conscientious personality trait. 
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Figure 14. Frequency of Conscientiousness Raw Scores 
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Figure 15. Distribution of Standardized Conscientiousness Scores 
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oe SHIP HANDLING EVALUATION SURVEY RESULTS 


Figure 16 1s a histogram that summarizes the overall participant expert ship- 
handler’s preference for interaction. In general, the experts viewed the novice JO’s 
ability to interact with other entities as a relatively import criteria for their evaluation of 
the JO’s performance. 65% of all respondents felt that how the JO interacts is at least an 
important criterion for evaluation. Furthermore, these experts desired to coach their 
novices through the evolution via continuous input and feedback and their pnmarv 
measure of interaction is how well the novice JO maintained close verbal communication 


with the expert ship-handler coach. 
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Figure 16. Distribution of Interaction Responses 


Figure 17 is a histogram that summarizes the overall participant expert ship- 
handlers’ preference for communication. The majority of experts did not feel that the 
novice JO’s personal ability to communicate with the Replenishment ship was important 
to their evaluation of the JO’s performance. Most expert ship-handlers feel that someone 
other than the novice JO performing the UNREP should handle personal communications 
between the approach ship and replenishment ship. Only 17% of all participants 


expressed communication as an important criteria for UNREP performance evaluation. 
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Figure 17. Distribution of Communication Responses 


Figure 18 summarizes the participant expert ship-handlers’ preference for 
analytical input. In general, most participant expert ship-handlers believe that a must be 
able to efficiently receive and process analytical information. While all participants 


believed that the novice JO’s ability is at least somewhat important, 67% of all 


espondents felt that analvtical input 1s at least an important, if not critical component of 


evaluating the novice's performance. 
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Figure 18. Distribution of Analytical Input Responses 


Figure 19 summarizes the participant expert ship-handlers’ preference for the 
novice JO to understand and efficiently process sensory input. All participants believed 
that the novice JO’s ability to demonstrate an understanding of sensory input was at least 
somewhat important to the expert’s overall evaluation. 35% of all participant expert 
ship-handlers view reaction to sensory information as a critical component of a successful 
UNREP and use the JO’s response to sensory information as a major element of UNREP 


performance evaluation. 
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Figure 19. Distribution of Sensory Input Responses 


Figure 20 is a histogram that provides a breakdown of how each individual 
participant expert ship-handler approaches UNREP. Figure 20 displays a uni-modal 
symmetric distribution with the majority of participant expert ship-handlers taking an 
attitude towards UNREP that is neither too flexible nor too stnct. In general, most 
evaluators allow some deviations from their execution preferences by the novice JO. 
Only 7% require the novice to perform the evolution exactly as the expert desires while 
only 5% of all participant experts allow the JO to perform the UNREP in any safe 
manner. 

The participant expert ship-handlers who tended towards a looser UNREP style 
placed less emphasis on time to perform the approach as a criterion for performance 
evaluation. In contrast, those experts who possess a more ngid UNREP style place more 


emphasis on time as an evaluation critenon. Regardless of UNREP stvle, most 


Un 
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participant expert ship-handlers believe that as expenence increases, time to complete the 
evolution will decrease. 

Approximately 50% of all participant expert ship-handlers believe that UNREP 
documentation and doctrine provides instruction that must be strictly adhered to. The 
other 50% of participant expert ship-handlers interpret UNREP documentation and 


doctrine as guidance that provides a flexible framework for execution. 
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Figure 20. Distribution of UNREP Styles 


D. OBSERVED CORRELATIONS 


Table 5 contains the intercorrelations observed between the five personality 
factors measured by the NEO-FFI. Table 6 contains the average intercorrelations for the 
NEO-FFI [MCCR92]. In general, the personality traits were more intercorrelated for the 
participant expert ship-handlers than for the average NEQ-FFI participant. The only 
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observed exceptions where the participants possessed lower than average intercorrelations 


were Neuroticism / Conscientiousness, Extraversion / Openness, and Agreeableness / 


Conscientiousness. 
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Table 5. Observed NEO-FFI Intercorrelations for 
Participant Expert Ship-Handlers. 
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Table 6. NEO-FFI Intercorrelations for 
Average Adults from the NEO-FFI Manual [MCCR92]. 
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In general statistics, a large correlation exists if the correlation coefficient (p) is 
greater than or equal to 0.8 and a small correlation exists if p is less than or equal to 0.5. 
However, the intricacies of human behavior make analyzing real people more art than 
science. Single sample t-Test with a probability of type I error (a) equal to 0.10 indicate 
that correlations with p greater than 0.10 are significant for the participant expert ship- 
handler sample [DEVO95]. Therefore, for the purposes of this research, any correlation 
greater than or equal to 0.10 was identified as a potentially significant correlation. 

Figure 21 graphically displays the calculated correlations between observed expert 
ship-handler Neuroticism values and measured UNREP evaluation criteria. In general, 
correlations were small in magnitude. All observed correlations were negative with the 
exception of the correlation between Neuroticism and Communication, which was only 
slightly positive. 

Figure 22 contains information describing the calculated correlations between 
expert ship-handler Extraversion values and measured UNREP evaluation criteria. In 
contrast with the Neuroticism correlations, most Extraversion correlations were positive 
with the exception of the correlation between Extraversion and UNREP style. The only 
apparent significant correlation was observed between Extraversion and Sensory Input (p 
= .18). This relationship could possible be explained by Sensory Input satisfying the 


extroverts need for stimulation from his surroundings [MCCR92]. 
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Figure 21. Correlations between Neuroticism and UNREP 
Evaluation Criteria for Participant Expert Ship-Handlers 
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Figure 22. Correlations between Extraversion and UNREP 
Evaluation Criteria for Participant Expert Ship-Handlers 
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Figure 23 highlights the correlations calculated between the observed expert ship- 
handler Openness personality characteristic and the measured UNREP evaluation criteria. 
All calculated correlations for Openness were positive. Furthermore, all correlations 
were only slightly positive with the exception of the correlation between Openness and 
Analytical Input (p = .16) and Openness and UNREP Style (p = .16). This result might 
be related to the modest association between Openness and measured intelligence since 
Analytical Input is more calculation based. Therefore, it is possible that an expert that is 
higher in Openness would tend to more analytical or cognitive intensive methods. 
However, it must be clear that Openness is not a measure of analytical sense or actual 
intelligence [MCCR92]. 

Figure 24 delineates the correlations calculated between the observed expert ship- 
handler Agreeableness personality characteristic and the measured UNREP evaluation 
criteria. All calculated correlations were positive with the exception of the correlation 
between Agreeableness and Communication, which was slightly negative. The only 
apparent significant correlation was observed between Agreeableness and UNREP Style 
(9 = .16). A potential explanation for this correlation is that the highly agreeable person 
is sympathetic to others and eager to help, resulting in an UNREP style that is more 
malleable to the trainee novice [MCCR92]. The expert who scores high in Agreeableness 
sees the UNREP as a training evolution for the novice and therefore has a less ngid 
UNREP style resulting in greater variability and less structure in order to let the novice 


learn from exploration. 
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Figure 23. Correlations between Openness and UNREP 
Evaluation Criteria for Participant Expert Ship-Handlers 
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Figure 24. Correlations between Agreeableness and UNREP 
Evaluation Criteria for Participant Expert Ship-Handlers 
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Figure 25 graphically displays the correlations calculated between the observed 
participant expert ship-handler Conscientiousness personality trait and the measured 
UNREP evaluation critena. Similar to the Openness correlations, all calculated 
Conscientiousness correlations were positive. However, two significant correlations 
existed between Conscientiousness and Analytical Input (p = .17) and Conscientiousness 
and Sensory Input (p = .39). The correlation between Conscientiousness and Sensory 
Input was the highest ied over the entire experiment. Experts who score high in 
Conscientiousness are usually purposeful and strong willed and are often associated with 
academic and occupational achievement [MCCR92]. These facts can be directly linked 
to the expert ship-handler’s priority to properly processing all forms of information input. 
The highly conscientious expert will not be able to ignore any information that may 


affect the UNREP, regardless of source. 
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Figure 25. Correlations between Conscientiousness and UNREP 
Evaluation Criteria for Participant Expert Ship-Handlers 
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Correlations were also calculated to determine if there was any relationship 
between the participant expert ship-handlers’ NEO-FFI personality traits and Tenney’s 
VCO profiles [TENN99]. Figure 26 illustrates the relationships between the measured 
participant personality traits and Tenney’s Passive VCO profile. Significant correlations 
existed between the Passive VCO profile and the Agreeableness (p = -.25) and 
Conscientiousness (p = -.11) personality traits. In direct contrast, Tenney’s Proactive 
VCO profile correlated with Neuroticism (p = -.20), Extraversion (p = .13), and Openness 


(9 = -.11). Figure 27 illustrates these correlations. 





Figure 26. Calculated Correlation Between Observed Participant Expert 
Ship-Handler NEO-FFI Personality Traits and 
Tenney’s Virtual Commanding Officer Passive Profile. 
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Figure 27. Calculated Correlation Between Observed Participant 
Expert Ship-Handler NEO-FFI Personality Traits and 
Tenney’s Virtual Commanding Officer Proactive Profile. 


E. DISCUSSION 


Due to the assistance of Vice Admiral Giffin and Vice Admiral Moore, 40 percent 
of all expert ship-handlers queried responded to the survey. This relatively high 
participation rate greatly increased the quality of the data package by producing relatively 
normal distributions for all areas analyzed. Indications are that the sample of participant 
expert ship-handlers is a decent approximation of the expert ship-handling population. 


Furthermore, the wide range of responses created a broader depth of elicitated ship- 
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handling knowledge that will directly translate into a better database for COVE 
programmers. 

Analysis of the participant expert ship-handler data package indicates that the 
surveyed experts possess some significant differences from the average adult. In general, 
experts were lower in Neuroticism, and higher in Extraversion and Conscientiousness. It 
is clear that whenever modeling an expert ship-handler, close attention needs to be 
focused towards the expert’s level of Neuroticism, Extraversion, and Conscientiousness 
to ensure that the expert behaves in a manner that is consistent with expertise. Future 
research will venfy that these personality characteristics apply to all experts regardless of 
the domain of expertise. 

It is important to note that there is still room for variability in an expert model’s 
personality. While the means for participant expert ship-handler levels of Neuroticism, 
Extraversion, and Conscientiousness were significantly different from the average adult 
population, Figure 5 illustrates that the standard deviations for the participant expert 
ship-handlers’ personality traits were very close to the average adult standard deviations. 
This indicates that experts can vary in personality just as much as the average person 
does, just within a different range. 

In addition to the model of expert personality, close attention should be focused 
towards the dynamic of expert evaluation of a novice. All ship-handling experts are 
going to evaluate novice performance based upon a unique combination of evaluation 
criteria. The evaluation criteria create a performance checklist that the novice is 
evaluated against and will consist of a combination of Interaction Responses, 


Communication Responses, Analytical Input Responses, Sensory Input Responses, and 
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the evaluator’s UNREP Style. The emphasis and pnority of this novice report card is 
related to the evaluator’s individual personality. Analytical Input will be stressed if the 
expert evaluator is relatively high in either Openness or Conscientiousness. Sensory 
Input will be a critical cnterion if the expert possesses a relatively large amount of 
Extraversion or Conscientiousness. Finally UNREP style will be a major performance 
criterion if the expert ship-handler performing the evaluation scores high in Openness or 
Agreeableness. 

Analysis of the data package also indicates that there are links between Tenney’s 
VCO profiles and the model of expert personality proposed in this research. Experts who 
are relatively low in Agreeableness and Conscientiousness will tend towards a more 
passive style of ship-handling as defined by Tenney’s previous work. Furthermore, and 
in direct contrast, expert ship-handlers who display relatively high Extraversion and 
relatively low Neuroticism and Openness will tend towards Tenney’s proactive VCO 
profile. 

With the proper composition of personality traits, and proper combination of 
evaluation cnteria and overall ship driving style, a relatively robust model of an 
experienced evaluator of ship-handling is developed. The model is robust because it 
covers a relatively wide range of personality inputs resulting in a potentially wide range 
of unique prototype expert ship-handlers. With proper development, the model born from 


this research could potentially grow into an accurate representation of human mentoring. 
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VI. CONCLUSIONS 


A. SUMMARY OF WORK 


Understanding how experts evaluate performance of tasks with large degrees of 
variability is a difficult undertaking. Just like the tasks themselves, the evaluation does 
not always have the same clear-cut criteria for each expert performing the evaluation. 
Often, critical evaluation criteria are as individualistic as the evaluating individual. 

The primary purpose of this thesis was to assess the relationship between an 
expert evaluator’s personality and the evaluation criteria the expert employed. COVE’s 
UNREP trainer, currently under development by NAWC-TSD, provided the ideal vehicle 
for this research since UNREP is a highly dynamic and challenging task performed by a 
novice whose performance evaluation is often a combination of unique objective and 
subjective criteria dependent upon the evaluating expert. 

In order to obtain information about the relationship between evaluation and 
personality, a review was performed on various topics including facets of personality, 
decision-making, individual differences, differences between experts and novices, and 
personality measurement. This review was a crucial step for selecting the proper 
personality assessment tool. Next, an experiment package was developed utilizing a 
questionnaire format in order to elicit information about personality, ship-handling style, 


and novice performance evaluation by a ship-handling expert. 
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After the experiment package was developed, the survey was administered to 
ship-handling experts resident in the fleet. Support from both Commander Surface Naval 
Forces Atlantic, Vice Admiral Giffin, and Commander Surface Naval Forces Pacific, 
Vice Admiral Moore was critical in ensuring adequate participation by busy ship- 
handling experts. With their help, 116 experts participated in the survey either via a 
paper based or web based version. 

After the end of the data collection period, the participant surveys were collated 
into a data analysis package. The data was analyzed for statistical summary information 
such aS means, minimum values, maximum values, mode, median, and standard 
deviations. The data was also analyzed for significant correlations existing between 
participant expert ship-handler personality traits and participant expert ship-handler 
evaluation criteria and ship driving style. Correlations were also analyzed between 
personality and Tenney’s VCO profiles. 

Understanding performance evaluation is a goliath work in progress. Results of 
this research, while beginning to explore the gray areas of variability in evaluation, will 
provide direct enhancement to the training potential of COVE by increasing the fidelity of 
COVE’s ITS. The insights gained from the expert ship-handler personality traits will 
increase the social realism of the ITS and therefore improve the manner of feedback given 
to the trainee since the ITS can now respond in a more human manner. Furthermore, the 
compilation of critical evaluation criterion and its relationship to personality will enhance 
the task training ability of COVE since it will allow the ITS to train in a manner similar to 


fleet experts. Finally, the creation of a ship-handling database for various fleet expert 
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ship-handlers will provide a solid footing for the overall future development and 


performance of COVE, increasing its quality as a training tool and its utility to the fleet. 


B. | THESIS QUESTIONS 


The following questions were addressed in this thesis: 


e Js there a relationship between one’s personality and one’s 
expertise? 
Analysis of the NEO-FFI personality inventory administered to the 
participant expert ship-handlers indicates that there is a relationship 
between personality and expertise. In general, inventoried experts 
were less neurotic and more extroverted and conscientious than the 
average adult. Inventoried experts possess levels of Openness and 
Agreeableness that do not deviate from expected norms for the average 


adult. 


e If such a relationship exists, can it be quantified? 


The model of an expert’s personality should possess the same levels of 
Openness and Agreeableness observed in the average adult, centering 
primarily around average with an approximately normal distribution. 


When modeling an expert’s personality, Neuroticism should range 
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from between very low to average with only a small percentage of 
experts possessing higher than average Neuroticism. In contrast, the 
expert personality model should possess levels of Extraversion 
primarily ranging between high and very high with a lower percentage 
of average extroverts. Finally, the expert personality model should 


possess a primarily average or high level of conscientiousness. 


e Does it extend beyond individual expertise to the expert’s 
evaluation of others’ performance? 
Analysis of the ship handling evaluation portion of the expert survey 
indicates that there is a relationship between personality and the 
evaluation of performance. Evaluation of the novice is based upon a 
combination of factors including proficiency of Interaction, 
Communication, processing of Analytical and Sensory Input, and level 
of adherence to the evaluator’s stylistic tendencies as displayed by the 
novice. The combination of these factors is umique for different 


experts and is dependent upon the expert’s personality profile. 


e What is the range of characteristics of different ship driving styles? 


The data collected during the ship-handling survey spans a wide range 
of ship-handling variability and will be collated and sent to NAWC- 


TSD COVE programmers for analysis. 
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The overall data gathered for this experiment serves as a basis for the 
programming of an ITS for COVE. With this research, a database now exists where a 
programmer can create a model of ship-handling expertise that is highly variable 
depending upon user input. If done correctly, the user could configure a personality 
profile that results in a VCO that interacts, evaluations, and provides feedback to the 
trainee in a way that is completely unique to the personality configuration. 

The relationship between personality, expertise, and evaluation proposed in this 
thesis will allow the ITS to perform in a more human-like manner, increasing the illusion 
of immersion for the trainee, and provide the trainee with accurate performance 
evaluation criteria, increasing the positive training transfer benefits. In addition to 
information about expertise and evaluation, specific information was gathered regarding 
preferences for acceptable UNREP performance. This information will aid programmers 
in determining acceptable limits and bounds for trainees to perform within. 

An important point is that this research is entirely limited to COVE and UNREP 
evolutions. While this research utilized surface ship-handling and COVE as a vehicle, 
the real road is how experts subjectively evaluate highly aggregate tasks such as UNREP. 
The theories and model of expertise provided by this research can potentially span across 
any domain of expertise, as long as the expertise is not composed of black and white 


levels of performance and evaluation. 
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C. RECOMMENDATIONS FOR FUTURE WORK 


Tenney’s initial research, combined with the efforts of this research, is a start for 
improving the man-machine interface of tomorrow’s computer based trainers. 
Continuance of this research could vector into numerous directions. Some possibilities of 


future work include: 


e Further investigation between Tenney’s VCO profiles and their 
relationship to an expert’s personality. 


e A further analysis of the data obtained from this experiment 
utilizing expert ship class as a variable 


e Examining the dynamic between expert and novice accounting for 
the personality of the expert as well as the personality of the 


novice. 


e A linear, or non-linear, regression model of evaluation with 
coefficients based upon the correlations proposed in this research. 


e An agent based autonomous agent, such as STEVE, that possesses 
a configurable personality within the bounds of expert personality 
as defined by this research. 
Understanding the intricacies of human behavior and its relationship to highly aggregate 
tasks that are subjectively evaluated is a mammoth effort. Hopefully, this is only the 
beginning of a series that attempts to quantify how humans act and react. Performance of 


this type of research is a critical step in increasing the utility of computers by making a 


computer more like a human instead of a human more like a computer. 
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APPENDIX A. COMMANDER, SURFACE NAVAL FORCES ATLANTIC 


ENDORSEMENT 





§ May 2000 
Dear Commanding Officers. 


The Naval Postgraduate School (NPS) maintains 2 proud 
tradition of serving as a “Tech Bridge”. accommodating advanced 
research In direct support of Navy goals and future development. 
The Conning Officer Virtual Environment (COVE) program is a 
prime example of how NPS research strives to take Technical back 
to Tactical. Cove 1s a surface ship-handling traincr that utilizes 
cutting edge computer graphics and physically based modeling to 
simulate the ship driving experience. COVE is scheduled for 
imminent release to SWOS and the fleet. CA Wen has) 


The enclosed survey Is part of an NPS thesis that directly | 
supports COVE. The survey seeks to increase the fidelity and 
realism of interactions between the COVE trainer and the student. 
By completing this survey. vour time will improve the quality of 
this 21“ century readiness tool. 

= _ a 
Vos ie Rube Se Sincerely. 


NESS pwd shod <\ S - 
SO. _* HENRY C. GIFFIN I] 
Cos, \ . Vice Admiral. U.S. Navy 
Dre Aaah avd Yaad dt Rate Sant, Aa buh 
ae - — < 


\ 
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ENDORSEMENT 
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COMMANDER NAVAL SURFACE FORCE 
U.S. PACIFIC FLEET 


12 Mav 2000 
Dear Commanding Officer. 


Naval Postgraduate School (NPS) maintains a proud tradition of 
Serving as a “Tech Bridge.” accommodating advanced research in direct 
3 support of Navy goals and future development. The Conning Officer 
Virtual Environment (COVE) program is a prime example of how NPS 
| research strives to take technical back to tactical. COVE 1s a surface ship- 
handling trainer that utilizes cutting edge computer graphics and 


phy sically based modeling to simulate the ship driving ce COVE 
is scheduled for imminent release to SWOS and the fleet. Fao Tore Ja = 
eee 


The enclosed survey is part of a NPS thesis that directly supports 
COVE. The survey seeks to increase the fidelity and realism of 
interactions between the COVE trainer and the student. By completing | 
this survey. your time will improve the quality of this 21" century | 
readiness preparation tool. | 


<> - 1aan ee 
Det OF * nn Ke 
 nasatice “somel science ARD MOORE. JR. 
: Py ae a Some “pe. Vice Admiral. U.S. Navy 
oe Ys LAA Ps 
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XC. EXAMPLE ADDENDUM TO INSTRUCTIONS 





enters nanos 





to VOMMAND 5 OF IER LSS BONHOMME RICHARD LHD 
os o INSYRUCTIONS FOR ENPERT SHIPHANDLER SURVEY 


Se hurVe. .» ICM Aharae On the iiiterset. You are c: wOUTELCE 10 Laricipary wn Ihe WED O poe Wa aia Ofte 6 


: pean See Fees Cy ieses fact ‘2 ge res ‘ oe hv ge ys i . $ ? a5 , 
Oy Nping ure fohowing accross ms the Acdres, Bar vinteenc: Expiorer Figure i, or ita the Location Bar 


. oS . . 
Netscne Commun cator Frere 2, 
* . . t . foe ~ 
: wey WAVY, “Ds te ose eT OVEN \ pS sa ee ess 
° -_ 0 A tre eee, 


‘ 


rae ACCE SN CODE &@ SAI CLIN tO tie wea skews. Moves “ase xenyitovecs. 


ee ex raheg a 7 yo . t ee Tha Ay potas “ q Fe ees eau! ’ 
be NOR “BOON tO Pee MS NS ns WD > based vers; Of. SON aS NO Fears tO retry ae Susy CV RACAC. 


ae : ways nor ‘ver * ant aw ve: 4 
Resurcicss Of (ONAL Picuse Cemale:e Ihe samc) Noster nan July 7. 2600 


7g coca oe sens lee Te TT 
; 





ee i = P= - ww mo = ov - 2 
sO xy eg a aa ee ale ae a 
Reet eae wee amin heme ergee weeemat % a ee rio: Mimcewe Kratos ae Ya -.~* 
~ -- ‘om NASR esem ~ 
Pepe ee tae Ro ome aw iw? 
ea eta 
=> 
Reh’, POLE RAIA TE ECHOCH, = 
SURVEY OF HOW ExPEEA 
Tat P MAND ORS CVALUATI 
NOVICES 
= 
ww 
dS . 
eq r~ . 
5 wer? Lege de” 7 


2 o ste 
7 ONS. LE 
tf ¢ ore pet Ns ior Cd 
am aehewi tome & a es 
so pore 
le ng see 
eee ete He wo) i De weet 2) pe amen Rae Ege 
Te gtr ren Gere 4 te Pad mE Wy wer tereee af Pe OYE 


Oe Re OR wee eet aes | oe eae ee Cut eeey ee 
Prepares 62 ate Mayle 4 mee cnet A Gam ene ey 


A men lat 


a a a It 







Cd 
Oh Reena, ee SRA epee, : ORNS center, ae ete OR ER tI 
. we LN ari PO NOS one rate Gna ait + "gy - 
Si ee 1A 0 PAS OS 
ite 7 OW at rene ot a " * - . _ = . des a - A 
we o4 ons % an we 
we wT Rae De we Ne rrr 
we “5 Meret eee weet —/ 
Wey 6 wee 
es m § 


ae 


ate 


SURVEY Of aoe EXPER” 
SHIP HANDLERS CVALLATS 
NOVICES 


©) 


“oy, 
og 


Psereere 2 
Exampic Tie Sezeen as viewed by 
eiscape Navigator 


4% 


(G0) 


® 
rege St, a cree 
TO Per 


Pease ate Pe OCKOE COON mapesers ats yer oulle Pactage 
Tom ec qeree ee we gates Cabet we He ey EEE OF Hee Gas ey 
oe ovimiamine® enter Shep caste etvnewe Gat arty Soe 
Preiittie ong? ate ere ger tevgete © Pee Oe, 


LOIRESACERSST ALS ESSas ERE? 1902 


e 
G 
< 
7 RO GAISE TETAS Ut 


te ae wine seneane SOA ST GOO UIT IE ARE ER IY GPT LITE ETT OT 5 ee woot yeicen, 


| 


Verw Respesnaly Suomsicd. 
Tt. Chrys Buzax 


Ie 


THIS PAGE INTENTIONALLY LEFT BLANK 


76 


en te An armen eny a we 


APPENDIX D. EXPERT SHIP-HANDLING EVALUATION SURVEY 


NAVAL POSTGRADUATE SCHOOL 


SURVEY OF HOW EXPERT SHIP HANDLERS 


EVALUATE NOVICES 





fosis RADUA oe 








eet 
: Ae wp (eae 
wo o. ~ o “ 
- ae pauls, P: fot San ce nn. 
gg ttl en pat ee = 
a ee a ! a 
on - = : 
ad ad \ 


ig 





a NAVAL POSTGRADUATE SCHOOL 


| “a ': SURVEY OF HOW EXPERT | 
3{ SA \e SHIP HANDLERS —e- fe 


wes EVALUATE NOVICES 


INTRODUCTION rrovided tis use is compauble with the purpow 
for whieh the snformution was collect! 

Why Should | Participate? By parmicipaanz in ws survey. vou 
acknowledge that vas understand that your 
OMvacy I Maite as stared shove. 


Bates upon sour capericence Gh wusrent a. 
a a om i ‘urthermere, vou are acknowledging that your 


awd 


oostiion, vou were identified as un cinert snp 2 5 Ceci: Cee 
ee ia = ee ied eK, PAMICIPIWUOR Is COMPucieiy VOUNIAMy. 
nandler. Your parucipaaon with this surety wail 


sie TiAl Curtain Bsaecctis Of Ssmucig: Woiuns 
are roalisac and permanent and that simulator. 
trarmine systems ure Capable of inntructac INSTRUCTIONS 
tramees in 2 simular tashion to how vou imstruct 
vour Junior OF ers. 


How To Complete This Survey 


COVE Simulator We know that vour ame 1s valuable. This 
survey should take epproumatcly 30 minutes to i 
ve os 7 PR lecee Po » agert ae $ 
TR SESEN supports. devclooment ob che compicte The survey consists of chroe parts j 
Conmag Orfour Virtual Environmen: (COVE). sol gees , ohi 
| one Psotc emoz ~ Seco ' 
stupedriving saamulaior sheduled for imminent A p pe at 
telease to both SWOS and the Neer COVE is 3 a a 
3» UNREP evdiuation section, 


tully tandec taming projec: being developed bs 
NPS. Nuxal Aw Wartere Ceater Traming 


mr “AWCTSD,. <a : Comple of this sur me cece S 
Systems Division (NAWCTSD,, and the Orfice Completion of this survey will require you 


of Naval Research TONR) to enther fill in blank response fields or compictc 
COVE allows jemor stip drivers the 2 muinple choice Ds circkne sour answer. 


Please answer every question in this survey to 
the best of vour ability. If there are ans 
problems with this survey, please contact: 


OOpONMUNTY tO practice dangerous ship handling 
evolutions on a shere-based faciity with no 
danges to mater. or personne!. 


LT. Carns Buziak. USN 
Code 32, Naval Postgraduate Schoo! 
{ 
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Privacy Information €93 Deer Koad. Rm 408 
Monterey. CA 93943-8006 
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| throughout this Survey. and sour pmvacy 1s DSN 878-3579 
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informmion <cOmemnec a2 the <¢xocament 
deser:bed herein will be retained permanently at Thank you for your ume 2nd paruer:pation. 
the Naval Postgraduate School or by higher 
authonn. Answers pronded by = your 


participanon will be used only for statistical 
analysis by the Departments of the Navy and 
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PROFESSIONAL DEMOGRAPHICS 








Rank: - 

Surtace 

Primary Warfare Specialty: Aviaton 

fclrele one) 

Submarine | 
Other 
Oe ee eee eee 
| 
Sex: Male | 
fertcie ane) Cantile 





Total Years of Officer Service: 


Number of years 1n a sea billet: 


I {Total} | 
I 


| Number of Commanding Officers you 
| ___ have served under while at sea: ! 





Current Sea Billet: | 

_ _ tat shore. fill in SHORE) 
| 

| 
! | Current Class of Ship: | 
Of at shore. fill in SHORE) i 
: Time since last at sea: | | 


(Months) 


0-50 | 

Approximate Number | 51 - 100 | 

of UNREP Approaches - 101 - 200 

(Total dunne your career): | | 
(circle onr) 200 - 299 
f j 

i 

More Than 300 
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PERSONALITY INVENTORY 
instructions 


This section of the survey consists of a short, 60 question, personality inventory. This personality inventory is a 
standardized professional psychological tool that NPS is utilizing to learn more about the experts residing in the 
fleet. NPS belicves that accurate modeling of expert leadership will enhance the fidelity of the COVE tranner. 
The inventory takes approximately £0 minutes to complete. Please answer all of the following 60 questions as 
accurately and truthfully as possible selecung an answer that best represents your opinion Try not to read too 
much into any question. Your first impression or thought after reading the question should be your answer. Do 
not go back and change a previous answer. Possible answers range from strongly disagree in the leftmost 
column to strongly agree in the nghtmost column. 


(SD) Strongly Disagree oc the statement ws defintely false 

(0) Disagree xf the statement ts mostly taise 

(N) Neutral Ul ycu cannot decade, of the statement 1s equally uve and false 
(A) Agree om the statement is mostly wue 

(SA) Strongly Agree ox the sateen is dchmiely rue 


Choose only one answer per question. There is no time limit. 
Inventory 
Question SD D N/A /SA; 


1. lam nots woma 


iw 


. Pike to have a lot of people around me 

3. Fdunibke lowasieumedaydreammg = —_ 
4. Ttry to be courtcous to everyone I meet | 
5. Lkeep my belongings clean anc ncat | 
6 Toften feel inferior to others. ee | 
7, Tlaugh casaly. 

3. Once I find the nght wav to do something, I sick to 1. : 

9. | often get inte arguments with my famaly and coworkers. 

10. Um peetiv good about pacing myscl} 50 as to get Unngs donc on ume. 

1) When Tm under a great dal of stress. someu mes I fect hike I’m going to pieces. 


12. I dont consider myself especially “light-hearted”. a | 


13. Tum intngued by the pauterns IF find m art and mun. 


14. Some people think I'm selfish and egoustical. Sen | 
15. lam nota very methodical perscon. hn | eee . 
16, Tray feet alone FB (eee 2 


17. Ereally enoy talking to pcopk. i 
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(SQ) Strongly Disagree ox the staan is definately false 

(0) ODfsagree if the scaemeno: is mosuy false 

(N) Neutral if vou cannoli decide, ur the sumtenent is equally trux acd false 
{A}  Agreece che statement is moscdy true 

{SA) Strongly Agree ar the staivnrcut s definitely uve 





19. | would rather ovoperale with othety thun compete with them. Sale 
eer c ee ers eee 


23. Pocry has ine or nv effevt of me. 


2A. [tend to be cynical and skepecal of ovhers’ imentiona. | 
25. These a clear vt of poals und work toward them tn an orderly fashion. oe 


26 Sometimes I iccl completely surthigs. 





ZT. Ternatly peefer to do things alone. 


28. Loften ay new and foreign fonds. alte 
29. Ubelieve thar most people will make advarniage of you if yoo Jet them poets 


30. ' waste a kt of tume before settling down to work 





31. trarcly Lect fearful or wwueus. 


32 Lutten fer] as if i'm bursung wah energy. 


34. Mum people | imow like me. 
3s ee Rooye my gvuls. 


36. Tuften get angry at the way people reat me. 


3. tama cheer eh porson a 
38 Thebeve we should look w gor celigious authorioes for decisions on moral issues, a 2 


39. Sowec people thauk of aw ay Gul] and Galculaking. 





40. When I make a comentioent. [ can always be counzed on 10 folk through. 





4]. Loo often, when things go wrong, I get dhscouraged and teri hke giving up. eo oe a 
fatmonaetmine TET 


43. Soroctrmes when | am reading poetry ar looking at a work of art, I feel a chill or wave of  ] 
caocicanni. ‘ 
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(SO) Strongly Disagree or the suwmeni ss definttcly false 

(D) Oisegree if the statenent is mostly false 

(N) Neutra? if vou cannot decide, of the statement 1s equally ire and false 
(A) Agree or the stat ment 1s mostly wruc 

(SA) Strongly Agree or the statement 1s definately ruc 





Question 'sp| DiN | A |SA| 


ee Carer 1 


45. Someumes I'm not as dependable of reliable as | should be. ! | 


46 Tam selduia sad o& depressed 1 | 


i 
! 


48 DPhaove bile ucest in speculanng on the nature of the universe of the humana condiboe. | 


: ! 
49° I generally ury to be thoughiful and considerate. | 


$9. Tama producuve person who always gets the job donc. 


$1. 1 often fect helpless and want someone else to solve my prodlems, | | 


$2 f ama sery active person ' 


$3. [have a lot of mtelloctual cunosty | 
S40 If T dont hke peopie, I tet Wer imow 1. ee | 


$5. I never seem to be able to gel orgamzed 






$6. At umes [ bave been so ashamed | yust wast to hide. 


' 
$7. | would rather go ary own way thn he a leader of mhers 


oe T often enjoy playing woth theenes or abstract ideas. 





$9 If necessary, Lam willmng (0 manipulate peopic to get what I want. 


60 Istnve fer excellenc: in everytung | do. nee 
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UNREP EVALUATION SURVEY 


Introduction 

This portion of the survey deals with how you would evaluate a junior ship handler. For the purpose of this 
survey, imagine that you have just supervised a young, less expenenced junior officer who has completed the 
approach on an AE. Following completion of the evolution, you are tasked with evaluating the JOs 
pertormance and providing the JO with feedback about the execution of the approach. This portion of the 
survey is intended to determine the specific criteria you use to evaluate a conning officer's UNREP approach 
performance. 


This inventory consist of 3 parts. While the entire inventory requires only about !5 minutes. there 1s no ume 
limit. Please answer each of the following questions considenng yeur requirements for a successful approach. 


Part | 


Part | requires you to evaluate each of the following 3 UNREP approach templates. Evaluation cntenia for 
each of the three UNREP styles are: 


Preterred - Ts method most closely maiches how you expect your Conmng Officer to make the UNREP approach. 

Acceptable - ‘This method may no be cxactly bow vou expect the approach to occur, bul you have no objections (o this type of 
execuuon 

Unacceptabie - You will net allow 2 Conning Officer to execute the UNREP in uns fashion. 


Legerd For Diagrams 
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ERFAKAWAY RICOMEMFND IN Q 


@Quume FIESIFARAWKY 2LFrOREIMOE yy 


CLOSE RECCEMEND Un 





1000 feet 
1800 feet 
2000 feet 
| 
| (ewcle one) 
Preferred Acceptable Unacceptable 
i 
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120 feet 


BREAKKWAY ECOMMEND DH 2 


‘Game PIREERFARAUAY ReccAmeNDn 


DEW RECOMMEND 2m 


1800 feet 


2000 feet 


{circle one} 


Preferred Acceptable 
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Unacceptable 


1000 feet 


1800 


2000 feet 


Preferred 


120 feet 


BLEARAWAY LECOMMEFND OF H, 
FRE-BIEAKAGHRY XN CHMEND Mh 


ALOWUGODS ReVCED in 





(circie Gav) 


Acceptable 
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Unacceptable 


Part Ii 


Part Il covers specific criteria that may be used to evaluate a Conning Officer Possible answers for each of the 


following questions in Part Il are: 


Not Applicable (N/A) -There 1s 90 need to perform this acoon 

Applicable (A) - Tus ts a relauvely minor action with large room for varuitivn of exsecubon. 

Somewhat important (SI) - An acuon uhat must be perforined to have « successful UNREP approach. bul 
wit) som: room for vanauon of execuuen. 

important (1) - This acuon must be performed well to have a successful approach with hitie vanaton of 
CXCCUlIORL 

Cnrtical (C) - It is smpossibie to successfully complete an UNREP approach without performing Uns action 
flawlessly. 


Please answer each of the following questions by providing one of the above ratings. If your ship does not have 
a required piece of equipment to perform an action, answer as if the required gear was installed on your ship. 


Conning Officer | 
Preparations/Qualifications | 1 | 
While vou do not necessarily expect the Conning | N/A : A : SI | I 


Officer to perfarm these actions, shry are knowledge | | 
requirements for a good Conning Officer 


I. Possesses an ardepth knowledge of own slup's : | ' 
nuname males of acculcruuon/develerauon and otber a | 

handling charactensthes. 

2. Possesses an u-depth knowledge of handling 
charactenstics of auzthary snp (e.g. acceleration and 
turn rates). 


3. Possesses an understanding bow Own Ship’s 
‘ibranons correspond to speed and nudder commands 
4 Pomcanatingiosemecenvoneaaticen | | 
5. Possesses an understanding of wind force effects 
on ship's freeboard. 


6. Possesses understanding of depth of ocean effecis | 


{ ' 


3. Possexses understanding of topography of ocean 
floor effects. 


samen anmc cnn Ob mente 


ee re 


| 
& Possesses understanding of sea state effects. | | 


11. Possesses understanding of JA/Integrated ship's 
arcunt 


2) 


12. Possesses undersuanthng of radio beadsev/wallbe- 
tallac for mtcroa] commaunicziions. 


15. Possesses understanding of Badge to Budge 
handset 
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10 


Not Applicable (N/A) -Thete 19 po nevd to perform thy acbon, 

Applicable (A) - This is 9 relavely manor acnon with large room for variauon of cxcawsion. 

Somewhat important (SI) - An action that must be performed to have a successful UNRF P approach, but 
with some room for vanauon of cxecupon. 

Important (I) - This action must be performed wvil to have a successful approach with litle variaueo of 
eExecuven, 

Critical (C) - it ty impossible to successfully compicte an UNREP approach witbout performing this action 
flawlessly. 





Conning Officer 
Preparations/Qualifications 
While vou do not necessarily expect the Conning NWA A 


Officer to perporm these actions, they are knowledge | 
requirements far g good Corsing Officer | 


SI | I 






14 Possesses understanding of UHF. 


tS) Possesses undersuanding of cellular 
ComEd CauCD. 

18. Possesscs understanding of Lghis (for 
Communication}. 


17 Possesses understanding of semaphores. 


18. Possessws understanding of forma] decision aids 
t‘informasion boards, cherkiiere. flaw charts etc .) 


19. Possesses understanding of NWP procedure. | 


toe ae teeter 


sckamineecs fl cmcniasinds' lp eneavsaen 5 <annanas'4 Sacer asamopaalacraiedees tea/omasasised 


20. Possesses understamking of SOPs. 
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Not Applicable (N/A) -There ts po need (o perform this acuoa. 

Appticabie (A) - This is a reftanvely aunor acuon with large room for vanation of execunon 

Somewhat important (SI) - An achon that must be performed 10 have a successful UNREP approach, but 
with some room for vanation of execunon. 

important (1) - Ths acuon ims be performed well w have a sucvessful approach with bide vanabon of 


exocunon. 
Critical (C) - lt 1s umposmbie to successfully complete aa UNREP approech wilbuut pertormsng tts acucn 
flawlessly. 
Execution 
These are actions you expect a good Canmng Officer | Si i C 


to perform well dunny the UNREP approach 


| 

| 
[. Masntuned close verbal commumcation berween 
Conn and CO/XO dunng approach \ en | fi 
2 Marotained regular commumcanons bet wera own | 
stip and Auxilsary dunng approach. 
3. Displayed peoper usc of tact to face 
communicahons. ‘ 


4. Displayed proper use of J A/Integrated stup's a | 
arcu. 
S. Displayed peoper use of radio headset/wallac- i. | 
tallue for intemal Commumcahuns. 


Stapnetnre ans tnieeirarnase [| see. 


S. Displaved proper use of cellular counmumecaan, | 


a ea aa he ae | 
commupieasiony, ae oe 
Il. Displayed proper use of funmal decasion aids ae | : 
tinformanon boards, checklist charts rtc..). { 
13. Displayed peoper use of SOPs. | ea | i 


14. Displayed proper use of rules of chumb (non- | 
numeric, non-catcularion based). 


ter eaelaicn bed ale und). a ee 
_other calculauon based rule of thumb). 

16. Displayed proper use of an rorermal checklist or lL 

other systematic routine for comptenon of mulestancs. 








i 
: 
4 
3 
= 
3 
E 


+ ee ome 


aoe ne nmnermn = 
nae 


17. Displayed proper use of a scan rovnine for input. 
(8. Displayed proper visual observations of wake to 
* yudee relanve motion. 

19. Displayed proper visual observauos of relanve 

eTspective to pudge relative Moun 

20. Displayed peoper visual observanon of rate of 
closure to judge rclabye mouco, 

21. Displayed proper and omely use of GPS 
positian/sspced 


22. Displayed proper use of MOBOARD. 


23. Displayed proper aod omely use of Stadimctcr 
unformanon. 


24. Displayed proper and omely use of Radar range 
ofor mauon. 


ey poem 
nt 


— 
el titan A tt tt ti, Ett 


25. Displayed proper visual estimations of range. 


26. Displayed peopes aod amely use of laser 
nectioder wformaton 
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IZ 


Part ill 


Part Ill consists of some general questions about UNREP approaches. Answers for Part 11 consist of multiple 


choice and fill in the blank that best correspond with your expert style of ship driving. 


What ts your minimum allowable lateral separation? 
7 eve Me eee ern es ee eee 

What is your maximum allowable lateral separation? 
fin feet} 


What is your ideal approach speed differential? 


fin rote | 


What is your ideal distance behind the 


Auxiliary pnor to commencement of approach? 
fin sards; 





What is an adequate arnount of ume to perform an 
approach for a new Junior Officer? 
(circle one) 


What is an adequate amount of ame to perform an 
approach for an experienced Senior Officer? 


(circie one) 


How would you characterize your management of Conning 
Officers during an UNREP? 


(circle one) 


ee ee 
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Under S minutes 
Between 5 and 10 minutes 


Between 10 and 15 minutes 
Between 15 and 20 minuias 


Greater Than 20 minutes 
Time 1S a function of speed and destance 


Thera is no “adequate” amoum of ime 
Under § minutes 
Between 5 and 10 minutes 


Between 10 and 15 minutes 
Between 15 and 20 minutes 


Greater Than 20 minutes 
Time is a function of speed and distance 


There is no “adequate” amount of time 
} allow large variations in 
execution from my preferences 


| aliow some vanations in 
execution from my preferences 


lallow no vanatons in 
execution from my preferences 


et ee a mk 


CEPA mir? qnamen ap enprey mnacsoenng: ane cnad 1a ; 


NWP is a detailed recipe for UNREP 
Which statement best descnbes the NWP? | 


(ctrcie one) | 


NWP is only a loose framework for UNREP 


| Aconning officer who primanty uses rules of thumb, 
| Ship's instruments and other guidelines to make 
' dacisions? 
(someone who knows numbers atrd facts) 
Which are you most comfortable with: | 
(circle ene) | or 
| Aconning officer who pnmarily uses his instincts and 
| sensory esnmatons? 


CONCLUSION 


Thank you for taking your valuabic time to 
complete this survey. The goal of this research is to 
determine how to tailor feedback provided to a Junior 
Officer (JO) during training in a Virtual Environment 
(VE) ship handling training simulator such as 
NAWCTSD’s Conning Officer Virtual Environment 
(COVE). Tailored feedback allows a simulator to 
train a JO in a way that is compatible with the JO’s 
Commanding Officer's ship handling style. If a JO 
can receive feedback that is similar to the feedback 
he receives from his shipboard mentor, the JO will 
have more effective simulator training time and be 
more productive while at sea. 

If you would like to leave suggestions or 
comments. feel free to enter them on the back of this 
survey: 


For additional questions or comments about this 
research, please contact: 


oi 


someone who crives a ship ile a car on the highwea 


LT, Chris Byziak, USN’ 
Code 32, Naval Postgraduate School 
833 Dyer Road: Rm 404 
Monterey. CA 93943-5000 
(831) 6564679 DSN 878-4679 


Please encourage your expert peers to participate 
in this survey. Greater expert participation will 
improve the results and create a better product 
released to the fleet. 


* Email any questions to LT. Chris Buziak at 
buziak @cs.nps.navy.mil 


Enter Your Comments Here 








oe 


APPENDIX E. UNREP CHARACTERISTICS BASED UPON 


APPROACH SHIP CLASS 


SMALLEST OBSERVED | AVERAGE MINIMUM LARGEST OBSERVED 
MINIMUM ALLOWABLE | ALLOWABLE LATERAL | MINIMUM ALLOWABLE 
LATERAL SEPARATION SEPARATION LATERAL SEPARATION 


es 





Table 7. Minimum Allowable Lateral Separation 
Between Approach Ship and Replenishment Ship 
Based Upon Approach Ship Class 


SMALLEST OBSERVED | AVERAGE MAXIMUM | LARGEST OBSERVED 
MAXIMUM ALLOWABLE | ALLOWABLE LATERAL | MAXIMUM ALLOWABLE 
LATERAL SEPARATION SEPARATION LATERAL SEPARATION 

(FEET) (FEET) 





Table 8. Maximum Allowable Lateral Separation 
Between Approach Ship and Replenishment Ship 
Based Upon Approach Ship Class 


25 


SMALLEST OBSERVED | AVERAGE OBSERVED | LARGEST OBSERVED 
SPEED DIFFERENTIAL sr SPEED DIFEERENTIAL | SPEED DIFFERENTIAL 
(KNOTS) (KNOTS) 


CV/CVN 


DD/DDG 


FFG 
Transport Ship 





Table 9. Allowable Approach Speed Differential 
Between Approach Ship and Replenishment Ship 
Based Upon Approach Ship Class 





SMALLEST OBSERVED | AVERAGE OBSERVED | LARGEST OBSERVED 
STARTING DISTANCE | STARTING DISTANCE | STARTING DISTANCE 
(YARDS) (YARDS) (YARDS) 

___ 1000 
CV/CVN 1000 1800 2000 
DD/DDG 3000 
1500 
1800 


Table 10. Allowable Starting Distance For Approach 
Between Approach Ship and Replenishment Ship 
Based Upon Approach Ship Class 





94 


[ANNES89] 


[AUST97] 


[BARR91] 


[BERN94] 


[BOLD87] 


[CAIR96] 


[CHI88] 


LIST OF REFERENCES 


Annet, J. (1989). Training Skilled Performance. Acquisition and 
Performance of Cognitive Skills. 


Austin, E. J., Deary, I. J., & Gibson, G. J. (1997). Relationships 
Between Ability And Personality: Three Hypothesis Tested. 
Intelligence, 25, No. 1, 49-70. 


Barrick, M. R. & Mount, M. K. (1991). The ‘big five’ personality 
dimensions and job performance: A meta-analysis. Personnel 
Psychology, 44, No. 1, 1-26. 


Bernstein, D. A., Clarke-Stewart, A., Roy, E. J., Srull, T. K., & 
Wickens, C. D. (1994). Psychology (3™ ed.). Boston: Houghton 
Mifflin. 


Boldovici, J. A. (1987). Measuring Transfer in Military Settings. 
In S. M. Cormier & J. D. Hagman (Eds.), Transfer of Learning: 


Contemporary Research and Applications (pp. 239-260). San 
Diego, CA: Academic Press. 


Caird, J. K., (1996). Persistent Issues in the Application of Virtual 
Environment Systems to Training. JEEE Computer Society: 
Human Interaction with Complex Systems, 3, 124-132. 


Chi, M., Glaser, R., & Farr, M. J. (Eds.). (1988). The Nature of 
Expertise. Hillsdale: Erlbaum Associates. 


5 


[CLON96] 


[COOK99] 


[COHE96] 


[DEVO95] 


[DREY96] 


[FISK49] 


[FURN96] 


[GAO93] 


Cloninger, S. (1996). Personality: descnption, dynamics, and 
development. New York: W.H. Freeman and Company. 


Cook, R. D. & Weisberg, S., University of Minnesota, “Applied 
Regression Including Computing and Graphics,” 
http://www.stat.umn.edu/arc/index.html (HTML Document) 1999. 





Cohen, M. S., Freeman, J. T., Fallesen, J. J., Marvin, F. F., & 


Bresnick, T. A. (1996). Training Cntical Thinking Skills for 


Battlefield Situational Assessment: An Experimental Test. 
Alexandna: U.S. Army Research Institute for the Behavioral and 


Social Sciences. (Technical Report No. 1050). 


Devore, J. L. (1995). Probability and Statistics for Engineering 
and the Sciences. 4 ed. Pacific Grove, CA: Brooks/Cole 


Publishing Company. 


Dreyfus, H. L. (1996). Intuitive, Deliberative, and Calculative 
Models of Expert Performance. In Zsambok, C. E. & Klein, G. 
(1997). Naturalistic Decision Making. Mahwah, New Jersey: 
Lawrence Elbaum. 


Fiske, D. W. (1949). Consistency of the factorial structures of 
personality ratings from different sources. Journal of Abnormal 
Social Psychology, 44, 329-344. 


Furnham, A. (1996). The FIRO-B, the Learning Style 
Questionnaire, and the Five Factor Model. Journal of Social 
Behavior and Personality, 11, No. 2, 285-299. 


United States General Accounting Office: Program Evaluation and 
Methodology Division. (1993). Developing and Using 
Questionnaires. (GAO/PEMD Publication No. GAO/PEMD- 
10.1.7). Washington, DC: U.S. Government Printing Office. 


96 


[HESL96] 


[JOHN98] 


[MEAD99] 


[MICH88] 


[MCCR87] 


[MCCR89] 


[MCCR92] 


[MCCR97] 


Heslegrave, R. J., & Colvin, C. (1996). An Exploration of 
Psychological and Psychophysiological Measures as Predictors of 
Successful Performance Under Stress. Alexandra: U.S. Army 
Research Institute for the Behavioral and Social Sciences. 
(Technical Report No. 1035). 


Johnson, W. L. (1998) Pedagogical Agents. Marina Del Rey, CA: 
Center for Advanced Research in Technology for Education 
(CARTE) USC Information Sciences Institute. 


Mead, A. & Walker, B., Naval Air Warfare Training Systems 
Division, “COVE Trainer,” 


http://www.ntsc.navy.mil/Programs/Tech/Virtual/VETT/cove.htm 
(HTML Document) 1999. 


Michel, R. R., & Riedel, S. L. (1988). Effects of Expertise and 


Cognitive Style on Information Use in Tactical Decision Making. 
Alexandria: U.S. Army Research Institute for the Behavioral and 


Social Sciences. (Technical Report No. 806). 


McCrae, R. R., & Costa, P. T., Jr. (1987). Validation of the five- 
factor model of personality across instruments and observers. 
Journal of Personality and Social Psychology, 52, 81-90. 


McCrae, R. R., & Costa, P. T., Jr. (1989c). The structure of 
interpersonal traits: Wiggin’s circumplex and the five-factor 
model. Journal of Personality and Social Psychology, 56, 586- 
7 


McCrae, R. R., & Costa, P. T., Jr. (1992). NEO PI-R Professional 
Manual. Odessa, FL: Psychological Assessment Resources, Inc. 


McCrae, R. R., & Costa, P. T., Jr. (1997). Stability and change in 
personality assessment: the revised neo personality inventory in 
the year 2000. Journal of Personality Assessment, 68, 86-94. 


oe 


[NPS99] 


[RUST99] 


[SALG97] 


[SCHU90] 


[TENN99] 


[TETT91] 


[WIDI97] 


Naval Postgraduate School MOVES Technical Report 
NPS-MV00-01, Variability in Ship Handling Expertise as 
Explained by Personality and Cognitive Style by LT. Christopher 
Buziak, 1 September 1999. 


Rust, J. (1999). Discriminant validity of the “big five’ personality 
traits in employment settings. Social Behavior and Personality, 
27, 99-108. 


Salgado, J. F. (1997). The five factor model of personality and job 
performance in the European community. Journal of Applied 
Psychology, 82, 30-43. 


Schultz, D. P. (1990). Theories of Personality (4"" ed.). Pacific 
Grove: Brooks/Cole Publishing Company. 


Tenney, K. R. (1999). A Virtual Commanding Officer, Intelligent 


Tutor For the Underway Replenishment Ship-Handling Virtual 
Environment Simulator. Unpublished Master’s Thesis, Naval 


Postgraduate School, Monterey, California, United States of 
America. 


Tett, R. P., Jackson, D. N., & Rothstein, M. (1991). Personality 
measures as predictors of job performance: A meta-analytic 
review. Personnel Psychology, 44, No. 4, 703-742. 


Widiger, T. A., & Trull, T. J. (1997). Assessment of the five- 
factor model of personality. Journal of Personality Assessment, 
68, 228-250. 


98 


BIBLIOGRAPHY 


Chief of Naval Operations, Underway Replenishment, Naval Warfare Publication, 


NWP 4.01.4, August 1996. 


Crenshaw, R. S., Naval Shiphandling, Annapolis, Maryland: Naval Institute Press, 
hy). 


Stavridis, James, Watch Officers Guide, Annapolis, Maryland: Naval Institute 
Press, 1992. 


99 


THIS PAGE INTENTIONALLY LEFT BLANK 


100 


INITIAL DISTRIBUTION LIST 


Defense Technical Information Center 
8725 John J. Kingman Road, Ste 0944 
Ft. Belvoir, VA 22060-6218 


Dudley Knox Library 
Naval Postgraduate School 
411 Dyer Rd. 

Monterey, CA 93943-5101 


Michael Zyda. Code MV/Dr. 

Modeling, Virtual Environments, and Simulations 
Naval Postgraduate School 

Monterey, CA 93943-5000 


Rudolph P. Darken. Code MV/Dr. 

Modeling, Virtual Environments, and Simulations 
Naval Postgraduate School 

Monterey, CA 93943-5000 


Commander, Naval Surface Force Atlantic Fleet 
1430 Mitscher Ave. 
Norfolk, VA 23551-2494 


Commander, Naval Surface Force Pacific Fleet 
2841 Rendova Road 
San Diego, CA 92155-5490 


Chief of Naval Operations (N62) 
2000 Navy Pentagon 
Washington DC 20350-2000 


101 


tO 


10. 


Capt. Jay Kistler, USN (N6M) 
N6M 

2000 Navy Pentagon 

Room 4C445 

Washington DC 20350-2000 


George Phillips 

CNO, N6M1 

2000 Navy Pentagon 

Room 4C445 

Washington DC 20350-2000 


LCDR. Dylan Schmorrow 
Office of Naval Research (342) 
800 No. Quincy Street 
Arlington, VA 22217-5660 


Dave Munroe 

Surface Warfare Officers School Command 
446 Cushing Road 

Newport, RI 02841-1209 


National Simulation Center (NSC) 
ATTN: ATZL-NSC (Jerry Ham) 
410 Kearney Avenue --- Building 45 
Fort Leavenworth, KS 66027-1306 


Robert Breaux, Dr. 

Naval Air Warfare Center Training Systems Division 
12350 Research Pkwy 

Orlando, FL 32826-3224 


Thomas Brown, Dr. 
Psychological Associates 
1465 Carson Creek 
Canyon Lake, TX 78133 


102 


Barry Peterson. Code CS/Pb 

Modeling, Virtual Environments, and Simulations 
Naval Postgraduate School 

Monterey, CA 93943-5000 


LT. Chns Buziak 
6006 Windhaven 
San Antonio, TX. 78239 


103 





J _* : : r ; : >. ’ le vb 
aL e fp. CATE 7 2 so : 46 
Sor f 4avetinanty, 1S OO", 







3 Rotter Presa, Ph : £ 
Naval Aje Wertare Core, Townine-) ote | ree 

12350 Resuuch Pex-y 

Oslasalo, FE. 32226-2224 









60 DOU 0 


6/02 29527-200 











en 


~~ 
1 


LIBRHAH 


INN 


KNO 


LE 


DUT 


> eto) 
be oe 
a 


/ 


8 00402741 


6 


3 


—— > 
————— C\) 


pombenl os) = ey 
mR 


eet aed 
a 
alates ke Tt 


Oe 
ae 


oe 
pia hee 
eet et 


alee oe 
eld eek tt ae pa 


a 
ae 


We Thais teen 


: me 1p 4 


men Ci bo 
ett ere 
a beatae oe re 
a 
eaeeet Loe 
Sie ee 


rire al 
tata 


ae rs 


betes ts 


hie 
oe 


Set a ae 
ba at 


lath tet 


phate ee Ses “f 
Saad) 0 


Pas 
ee ees 


$4 Tee 
ieee 


TS 
Wee a4o ow, 
td 


aa Tats 
- 


ed 
aed eo ae 


=. ae 


Ln 
2. 
hd es | ee 


| 
lie 
a 
oes a > 
bit eee 


em aeons 
eee 
on 


babes Oe 
5 ier Sc eee 


> 

y - ul SNC aa eps 

tL to ered ; ake Er 

aa 4 aad 

te oe een eos 7 
ba SION 


CE es 
LY 


aoe 
= a 
oe are ; : 
Se 


Lie Peseta = 
Dalal a ee ace 
aed 
et a 


Ca Cena 
este 


petted et 


=! 
- MA oe 


wtemets 


edi CPs 
mei are Gacy 


Sa 
rel eee 
nite Le 
etary 


ad ae 
or 

id ie 
ae 


p 
bated eT 
ade e 


Dual 
ie 
oe ery 


abet) vet 
hidesieetiiel oe tt) 


Peete 
re ae 


be a ar er 
Ded SLL nes 
Cots 


bidder 
oe Oy Bee 
hte kt 


ert 
rT Oe Fy 
bbe Sr es 


ue < : , oe 

tt : : i 

ae hae 
es 

Pe = 

F%en 8 

Cae 


he ot 
S 


Cr 
4 a. ‘ 
Haha, det Es 
pedal ers 
ae 


od 
erie ge 
oe 


rexste as 
A} or) 


het ae 
Sa a eae 
ae oot are 


on) 


een8 
ett Lee 


a 


ey 


LS 
a 


Seiisee ay 


ae CL 


Cen 
Cet 


Nideta rol rer 

“Areas sesdy 

ae} * 

ee ‘ 

oe eee y Pie ee 


A eres 
res 


ps a 
I 


atl 
Seen 


~~ wee 
Le) re 


we. 
tiled boll P} Lee er 2 


to 
kt 


ft 


Cet ed 
ete 


coe 
LLY ery 
Ad 


Lp diel 


rr 
| ae 
iP a 


Ca et Tt 


ba 


ao 
bed tet Dae 
ous 


ee ee 
hl 


ihe tT 


ial 
088 ge 


nee eee Se a 
gland pao 
deh ld ee > 


“a%ey 


7 


eT ae 


belied LLP me 


a tee & 


etal CL ne 
ead LSS oe 
LL ea 
. » 


ad oy 
per) 
aod 
-: Pane 
: 


ae 


a 
as 


Siete ee 
Lam eee re) 
Lets rt 


ar) 
“Aarts Bhs wy y a 


Pe en Pee 
Lie tT oy 


va at | 


ar 
een ye 


Bie 
ite thd | 
ith ee ee 


ea Car 
TTY 
Steed Tee te Ot at, 


are 
babe i tas TL 


Seid het, 


on 
he ees 
Seah he ead 


i Tt 
ale a ed pe 


ne TTT) 
bated ta mee + iat 
bite ode fah a 
ed ae errs 
Lae. oe 


a 
SE rie 
dea ata’ 


mck gk fete tthe 
Td tae oy ed 


oe SET 
tel bell ere 
5 oo tats see eee 
Lal to 


Du or) 
pact tS eal 


iether he fe 
ed 


moet ded Se Pa 
ss 4a Ri oe, Bayar tenta) 
ae Yet hOy oe gy ir ake 
Sorat t bebe id Lt Th 

obec Ths f 





ad 7 


(seas 
ers 


term iets 


iis adel 


ahs eet ee 


ete | 
rire) 
lee} 


gore ve 

at ett 
ead FP ke gy 
bt dehb te eres 
ree 


